Gitlab runner is spamming VM with docker volumes

I use the runner with follwoing CI/CD Pipeline.

variables:
  GIT_CLONE_PATH: $CI_BUILDS_DIR/$CI_CONCURRENT_ID/$CI_PROJECT_NAME

stages:
  - build

build-job:
  stage: build
  cache:
    - key:
        files:
          - yarn.lock
      paths:
        - ./webapp/.yarn-cache/  
  image: harbor.company.local/cache_dockerhub/library/node:16-alpine3.16
  script:
    - cd webapp
    - which yarn || npm install -g yarn
    - yarn install
    - yarn build:internal-deps
  tags:
    - docker

test-job:
  stage: build
  script:
    - echo `pwd`
    - ls -la
  tags:
    - docker

After some runs the runner is reporting “No space left on device”. Checking the VM is see that the /var folder is full. Most data is stored in /usr and inside the docker volumes folder. How I can avoid this spamming of vöumes?

runner-jjynfhp-project-10-concurrent-0-cache-3c3f060a0374fc8bc39395164f415a70
runner-jjynfhp-project-10-concurrent-0-cache-c33bcaa1fd2c77edfc3893b41966cea8
runner-jjynfhp-project-11-concurrent-0-cache-3c3f060a0374fc8bc39395164f415a70
runner-jjynfhp-project-11-concurrent-0-cache-c33bcaa1fd2c77edfc3893b41966cea8
runner-jjynfhp-project-13-concurrent-0-cache-3c3f060a0374fc8bc39395164f415a70
runner-jjynfhp-project-13-concurrent-0-cache-c33bcaa1fd2c77edfc3893b41966cea8
runner-jjynfhp-project-5-concurrent-0-cache-3c3f060a0374fc8bc39395164f415a70

docker volume prune will not remove this data, so maybe all volumes are in use.
How I can avoid this behavior?

I’m also seeing this, but I’m quite sure I’ve seen it for non-gitlab volumes (on another machine) as well, so the root-cause might be an docker issue.

In my case I’ve verified that the volumes are not in use by any running (or stopped) containers using docker ps -a --filter 'volume=VOLUMENAME

The volumes are also “dangling” according to docker volume ls -qf dangling=true

The last command of jenkins - Docker volume cleanup of gitlab-runner - Stack Overflow removed the volumes for me.

volumes wont be deleted since update to docker 23 · Issue #4028 · docker/cli · GitHub looks like might be the root cause.

Fix pruning anon volume created from image config by cpuguy83 · Pull Request #45147 · moby/moby · GitHub might fix it without requiring pruning named volumes

Hi Ingo,

I’d say this is quite normal behavior - the root cause is cache

So all of those docker volumes are actually project caches, ones that you have defined in your .gitlab-ci.yml file:

cache:
    - key:
        files:
          - yarn.lock
      paths:
        - ./webapp/.yarn-cache/  

So, perhaps you need to review the way how you store/upload cache (maybe it’s too granulated). Perhaps you need a bit more storage on that VM (e.g. I have 256GB on my Runners, which seems to be perfectly fine for around 10ish projects).

In any case, make sure to maintain your GitLab & Runners regularly. docker system prune with a few different options is a good start. You can also clear Runner caches easier from UI on a Project level (CI/CD > Pipelines > Clear Runner Caches (top right corner). But normally 1x a month I just prune all dangling images and all volumes from the Runner.

Hope this helps!

While this is indeed expected, and Docker volumes are indeed used for cache by GitLab runners.

And docker system prune running is a good idea. There is also a command for only cleaning up unused Docker volumes via:

docker volume prune