Gitlab runner on Ubuntu: Docker out of space error

I am trying to run my own CI runner on AWS using Ubuntu 16.04.

I successfully followed the Gitlab instructions for installing docker and gitlab-ci-multi-runner and can start jobs on my new runner.

However, my jobs fail with docker reporting:

Error processing tar file(exit status 1): write /lib/udev/hwdb.bin: no space left on device

The only changes I’ve made to configuration are:

/etc/gitlab-runner/config.toml: set privileged to true rather than false
/etc/docker/daemon.json: {“storage-driver”:“overlay2”}

I’ve checked that I have sufficient space on Ubuntu and have not exhausted inodes on the filesystem.

I think I have a fairly standard setup so I’m hoping someone else has seen this before? Thanks

Can you check any form of syslog / docker logs?

And maybe try using overlay instead of ovelay2 just to rule that out?

Hi @stefanvangastel - thanks for taking the time to answer.

I ran the build whilst tailing the log and docker ran out of space with this message

Mar 28 17:55:07 ip-172-31-52-234 dockerd[1015]: time=“2017-03-28T17:55:07.504124209Z” level=error msg="Failed to log msg “\x1b[0m\x1b[91mnpm ERR! tar.unpack untar error /root/.npm/glob-parent/2.0.0/package.tgz” for logger json-file: write /var/lib/docker/containers/9603f33c5ea5102843467a0e01a92f079e3345a9c5fc074920cf5bdb4b7a3611/9603f33c5ea5102843467a0e01a92f079e3345a9c5fc074920cf5bdb4b7a3611-json.log: no spac

Poor thing didn’t even get the chance to write the “e” in the word “space”. I suppose this is a docker configuration issue? It isn’t an issue with disk space as I was keeping an eye on the volumes as the build ran.

The Ubuntu server is a t2.medium AWS server and I think has 4GB of RAM available to it.

Actually - this seems to be a simple issue with disk space

Running watch df -h I see the available space being rapidly consumed… I’m surprised it eats so much with a small build taking place

Every 2.0s: df -h Tue Mar 28 18:04:34 2017

Filesystem Size Used Avail Use% Mounted on
udev 2.0G 0 2.0G 0% /dev
tmpfs 396M 5.7M 390M 2% /run
/dev/xvda1 7.8G 6.1G 1.4G 83% /
tmpfs 2.0G 0 2.0G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 2.0G 0 2.0G 0% /sys/fs/cgroup
tmpfs 396M 0 396M 0% /run/user/1000
overlay 7.8G 6.1G 1.4G 83% /var/lib/docker/overlay/15ff109d087f039418e984f88cf14370fb5aefe09afa68fc83b669895dc2a8a0/merged
shm 64M 0 64M 0% /var/lib/docker/containers/31002535386fce5ca27ebe689d13e646b858ad18ff3c013c2c9985096f441fbc/shm
overlay 7.8G 6.1G 1.4G 83% /var/lib/docker/overlay/e931d849a701d26f933f7647fdd1d3def7439de2b898f378c5accd451b519c82/merged
shm 64M 0 64M 0% /var/lib/docker/containers/be476767e1e1cc01d621785cfcd0cc0ab9bbb5581308036f77adfd2f6db04d61/shm

Off to provision a much bigger volume on AWS. Thanks