No space left on device

GitLab CI/CD
1

Hello, we are having an issue where builds fail do to the error:

Running with gitlab-runner 10.8.0 (079aad9e)
  on Global Runner 54152402
Using Docker executor with image registry.liquidcrystalstudios.com/intercodes/docker-image-intercodes ...
Pulling docker image registry.liquidcrystalstudios.com/intercodes/docker-image-intercodes ...
Using docker image sha256:7dad615fb72a5843cddcd42932b30784dce0c719643270057d5086ef7db50948 for registry.liquidcrystalstudios.com/intercodes/docker-image-intercodes ...
ERROR: Job failed (system failure): Error response from daemon: mkdir /var/lib/docker/overlay2/b8f0297596a3ffa7e0a30d6a7a89de724e6e88495a94e17d7ec265c3a642d6ca-init: no space left on device

/var/lib/docker is a symlink to /mnt/gitlab/docker
We are not out of space or inodes

[root@ip-172-31-26-23 lib]# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/xvda1       10G  3.6G  6.5G  36% /
devtmpfs        3.8G     0  3.8G   0% /dev
tmpfs           3.9G  4.0K  3.9G   1% /dev/shm
tmpfs           3.9G   17M  3.9G   1% /run
tmpfs           3.9G     0  3.9G   0% /sys/fs/cgroup
/dev/xvdf       493G  185G  283G  40% /mnt/gitlab
tmpfs           782M     0  782M   0% /run/user/1000
[root@ip-172-31-26-23 lib]# df -i
Filesystem       Inodes   IUsed    IFree IUse% Mounted on
/dev/xvda1      5242240   69049  5173191    2% /
devtmpfs         994812     285   994527    1% /dev
tmpfs           1000955       2  1000953    1% /dev/shm
tmpfs           1000955     411  1000544    1% /run
tmpfs           1000955      16  1000939    1% /sys/fs/cgroup
/dev/xvdf      32768000 2783917 29984083    9% /mnt/gitlab
tmpfs           1000955       1  1000954    1% /run/user/1000
[root@ip-172-31-26-23 lib]#

Gitlab versions:

GitLab 11.6.3 (bc9391c)
GitLab Shell 8.4.3
GitLab Workhorse v7.6.0
GitLab API v4
GitLab Pages 1.3.1
Ruby 2.5.3p105
Rails 5.0.7
postgresql 9.6.11

This is preventing all CI pipelines from running. Does anyone have any ideas?

2

I’m am going to ping this. We are still having this issue on multiple project. We are fully up-to-date.

I would really appreciate some help on this.

3

Not sure if it’s exactly your case but you may try to check disk space and inodes consumption during the job process. I’ve seen cases when either was 2x during the pipeline run. Ie when gitlab trying to pull Docker image it downloads layers and unpacks it consuming the disk space. In case of low disk space (sort of “No space left on device…” ) it do some cleanup so once you check resources AFTER all the action you may not catch the exact spike.