Gitlab runner is spamming VM with docker volumes

GitLab CI/CD
,
1

I use the runner with follwoing CI/CD Pipeline.


variables:
  GIT_CLONE_PATH: $CI_BUILDS_DIR/$CI_CONCURRENT_ID/$CI_PROJECT_NAME

stages:
  - build

build-job:
  stage: build
  cache:
    - key:
        files:
          - yarn.lock
      paths:
        - ./webapp/.yarn-cache/  
  image: harbor.company.local/cache_dockerhub/library/node:16-alpine3.16
  script:
    - cd webapp
    - which yarn || npm install -g yarn
    - yarn install
    - yarn build:internal-deps
  tags:
    - docker

test-job:
  stage: build
  script:
    - echo `pwd`
    - ls -la
  tags:
    - docker

After some runs the runner is reporting ā€œNo space left on deviceā€. Checking the VM is see that the /var folder is full. Most data is stored in /usr and inside the docker volumes folder. How I can avoid this spamming of vƶumes?

runner-jjynfhp-project-10-concurrent-0-cache-3c3f060a0374fc8bc39395164f415a70
runner-jjynfhp-project-10-concurrent-0-cache-c33bcaa1fd2c77edfc3893b41966cea8
runner-jjynfhp-project-11-concurrent-0-cache-3c3f060a0374fc8bc39395164f415a70
runner-jjynfhp-project-11-concurrent-0-cache-c33bcaa1fd2c77edfc3893b41966cea8
runner-jjynfhp-project-13-concurrent-0-cache-3c3f060a0374fc8bc39395164f415a70
runner-jjynfhp-project-13-concurrent-0-cache-c33bcaa1fd2c77edfc3893b41966cea8
runner-jjynfhp-project-5-concurrent-0-cache-3c3f060a0374fc8bc39395164f415a70

docker volume prune will not remove this data, so maybe all volumes are in use.
How I can avoid this behavior?

2

Iā€™m also seeing this, but Iā€™m quite sure Iā€™ve seen it for non-gitlab volumes (on another machine) as well, so the root-cause might be an docker issue.

In my case Iā€™ve verified that the volumes are not in use by any running (or stopped) containers using docker ps -a --filter 'volume=VOLUMENAME

The volumes are also ā€œdanglingā€ according to docker volume ls -qf dangling=true

The last command of jenkins - Docker volume cleanup of gitlab-runner - Stack Overflow removed the volumes for me.

volumes wont be deleted since update to docker 23 Ā· Issue #4028 Ā· docker/cli Ā· GitHub looks like might be the root cause.

Fix pruning anon volume created from image config by cpuguy83 Ā· Pull Request #45147 Ā· moby/moby Ā· GitHub might fix it without requiring pruning named volumes

3

Hi Ingo,

Iā€™d say this is quite normal behavior - the root cause is cache :slight_smile:

So all of those docker volumes are actually project caches, ones that you have defined in your .gitlab-ci.yml file:

cache:
    - key:
        files:
          - yarn.lock
      paths:
        - ./webapp/.yarn-cache/

So, perhaps you need to review the way how you store/upload cache (maybe itā€™s too granulated). Perhaps you need a bit more storage on that VM (e.g. I have 256GB on my Runners, which seems to be perfectly fine for around 10ish projects).

In any case, make sure to maintain your GitLab & Runners regularly. docker system prune with a few different options is a good start. You can also clear Runner caches easier from UI on a Project level (CI/CD > Pipelines > Clear Runner Caches (top right corner). But normally 1x a month I just prune all dangling images and all volumes from the Runner.

Hope this helps!

1 Like