Gitlab ci: no space left on device, after update with lots of free disk

GitLab CI/CD
1

Hello, we just updated to gitlab 11.3.0. And now we are running into an issue where several of our projects won’t build due to ‘no space left on device’ errors in the build process. We are using the overlay fs driver. We have tried to use other drivers with the same issue.

Looking at the disks, there is lots of space free: (Taken during a build that fails)

Filesystem      Size  Used Avail Use% Mounted on
/dev/xvda1       10G  3.4G  6.7G  34% /
devtmpfs        3.8G     0  3.8G   0% /dev
tmpfs           3.9G  4.0K  3.9G   1% /dev/shm
tmpfs           3.9G   33M  3.8G   1% /run
tmpfs           3.9G     0  3.9G   0% /sys/fs/cgroup
/dev/xvdf       493G  169G  299G  37% /mnt/gitlab
tmpfs           783M     0  783M   0% /run/user/1000

All gitlab files are under /mnt/gitlab, which has 169 GB free. If we retry the builds, sometimes after failing 10 or so times it will succeed. This is running on a AWS t2.large on centos 7. I am at a loss as to why it would start failing, but I assume it has something to do with the update. If we reboot the server, the builds will work for at least one build.

Any ideas?

Thank you,
Trevor Flynn

2

Hello,

Can you use df -ih instead of df -h?

Also, check this please:

sysctl fs.inotify.max_user_watches

3

Hello, this is the result of df -ih taken during a build as it fails

[centos@ip-172-31-26-23 ~]$ sudo df -ih
Filesystem     Inodes IUsed IFree IUse% Mounted on
/dev/xvda1       5.0M   66K  5.0M    2% /
devtmpfs         972K   287  972K    1% /dev
tmpfs            978K     2  978K    1% /dev/shm
tmpfs            978K   432  978K    1% /run
tmpfs            978K    16  978K    1% /sys/fs/cgroup
/dev/xvdf         32M  2.5M   29M    8% /mnt/gitlab
overlay           32M  2.5M   29M    8% /mnt/gitlab/docker/overlay2/d5d7250dda4437d282e39210168a901f2deb64ba1cc55d1d9f87d61a7c4420d8/merged
shm              978K     1  978K    1% /mnt/gitlab/docker/containers/4cdb2de1ce5e31215c9eea7e6fc14963444ac3b85dc45645de7303689046470b/shm
tmpfs            978K     1  978K    1% /run/user/1000

And this is the sysctl command

[centos@ip-172-31-26-23 ~]$ sudo sysctl fs.inotify.max_user_watches
fs.inotify.max_user_watches = 8192
4

Hello,
We still do not have working CI, is there someone who might have insight into our problem?

Thank you,
Trevor Flynn

5

If anyone else comes across this issue, this is the fix for us: https://github.com/guard/listen/wiki/Increasing-the-amount-of-inotify-watchers
We increased the watchers.

Still note sure why this broke after the update. I hope someone finds this solution as useful as we did.