Gitlab runner is spamming VM with docker volumes

I use the runner with follwoing CI/CD Pipeline.


variables:
  GIT_CLONE_PATH: $CI_BUILDS_DIR/$CI_CONCURRENT_ID/$CI_PROJECT_NAME

stages:
  - build

build-job:
  stage: build
  cache:
    - key:
        files:
          - yarn.lock
      paths:
        - ./webapp/.yarn-cache/  
  image: harbor.company.local/cache_dockerhub/library/node:16-alpine3.16
  script:
    - cd webapp
    - which yarn || npm install -g yarn
    - yarn install
    - yarn build:internal-deps
  tags:
    - docker

test-job:
  stage: build
  script:
    - echo `pwd`
    - ls -la
  tags:
    - docker

After some runs the runner is reporting “No space left on device”. Checking the VM is see that the /var folder is full. Most data is stored in /usr and inside the docker volumes folder. How I can avoid this spamming of vöumes?

runner-jjynfhp-project-10-concurrent-0-cache-3c3f060a0374fc8bc39395164f415a70
runner-jjynfhp-project-10-concurrent-0-cache-c33bcaa1fd2c77edfc3893b41966cea8
runner-jjynfhp-project-11-concurrent-0-cache-3c3f060a0374fc8bc39395164f415a70
runner-jjynfhp-project-11-concurrent-0-cache-c33bcaa1fd2c77edfc3893b41966cea8
runner-jjynfhp-project-13-concurrent-0-cache-3c3f060a0374fc8bc39395164f415a70
runner-jjynfhp-project-13-concurrent-0-cache-c33bcaa1fd2c77edfc3893b41966cea8
runner-jjynfhp-project-5-concurrent-0-cache-3c3f060a0374fc8bc39395164f415a70

docker volume prune will not remove this data, so maybe all volumes are in use.
How I can avoid this behavior?

I’m also seeing this, but I’m quite sure I’ve seen it for non-gitlab volumes (on another machine) as well, so the root-cause might be an docker issue.

In my case I’ve verified that the volumes are not in use by any running (or stopped) containers using docker ps -a --filter 'volume=VOLUMENAME

The volumes are also “dangling” according to docker volume ls -qf dangling=true

The last command of jenkins - Docker volume cleanup of gitlab-runner - Stack Overflow removed the volumes for me.

volumes wont be deleted since update to docker 23 · Issue #4028 · docker/cli · GitHub looks like might be the root cause.

Fix pruning anon volume created from image config by cpuguy83 · Pull Request #45147 · moby/moby · GitHub might fix it without requiring pruning named volumes

Hi Ingo,

I’d say this is quite normal behavior - the root cause is cache :slight_smile:

So all of those docker volumes are actually project caches, ones that you have defined in your .gitlab-ci.yml file:

cache:
    - key:
        files:
          - yarn.lock
      paths:
        - ./webapp/.yarn-cache/  

So, perhaps you need to review the way how you store/upload cache (maybe it’s too granulated). Perhaps you need a bit more storage on that VM (e.g. I have 256GB on my Runners, which seems to be perfectly fine for around 10ish projects).

In any case, make sure to maintain your GitLab & Runners regularly. docker system prune with a few different options is a good start. You can also clear Runner caches easier from UI on a Project level (CI/CD > Pipelines > Clear Runner Caches (top right corner). But normally 1x a month I just prune all dangling images and all volumes from the Runner.

Hope this helps!

1 Like