Hello, we just updated to gitlab 11.3.0. And now we are running into an issue where several of our projects won’t build due to ‘no space left on device’ errors in the build process. We are using the overlay fs driver. We have tried to use other drivers with the same issue.
Looking at the disks, there is lots of space free: (Taken during a build that fails)
Filesystem Size Used Avail Use% Mounted on /dev/xvda1 10G 3.4G 6.7G 34% / devtmpfs 3.8G 0 3.8G 0% /dev tmpfs 3.9G 4.0K 3.9G 1% /dev/shm tmpfs 3.9G 33M 3.8G 1% /run tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup /dev/xvdf 493G 169G 299G 37% /mnt/gitlab tmpfs 783M 0 783M 0% /run/user/1000
All gitlab files are under /mnt/gitlab, which has 169 GB free. If we retry the builds, sometimes after failing 10 or so times it will succeed. This is running on a AWS t2.large on centos 7. I am at a loss as to why it would start failing, but I assume it has something to do with the update. If we reboot the server, the builds will work for at least one build.