Cleanup cache.zip

GitLab CI/CD
1

My gitlab runner ran out of disk space.

Investigating the server, I looked at the docker data, and located these volumes
One project seems to be occupying half of the runners disk space.

du -sm /data/docker/volumes/runner-bb4211e4-project-60-concurrent-*
371752	/data/docker/volumes/runner-bb4211e4-project-60-concurrent-0-cache-3c3f060a0374fc8bc39395164f415a70
36206	/data/docker/volumes/runner-bb4211e4-project-60-concurrent-0-cache-c33bcaa1fd2c77edfc3893b41966cea8
125795	/data/docker/volumes/runner-bb4211e4-project-60-concurrent-1-cache-3c3f060a0374fc8bc39395164f415a70
17233	/data/docker/volumes/runner-bb4211e4-project-60-concurrent-1-cache-c33bcaa1fd2c77edfc3893b41966cea8

Clearing runner caches from the GUI does not change anything

Next, looking into the volumes I find a number of cache.zip files

root@wptsrv2:/data/docker/volumes/runner-bb4211e4-project-60-concurrent-0-cache-3c3f060a0374fc8bc39395164f415a70/_data/awc500platform/oe-lite# ll build_*/cache.zip
-rw------- 1 root root 141560984222 Aug 19 08:30 build_ls1021a-1/cache.zip
-rw------- 1 root root   5709970405 Aug 20 15:50 build_ls1021a-2/cache.zip
-rw------- 1 root root   5710095061 Aug 26 18:23 build_ls1021a-3/cache.zip
-rw------- 1 root root 116792992440 Oct 29 15:15 build_ls1021a-5/cache.zip
-rw------- 1 root root  74686467701 Jan  3 10:43 build_ls1021a-6/cache.zip
-rw------- 1 root root  30547919402 Jan 20 18:53 build_ls1021a-7/cache.zip

And in another volume

root@wptsrv2:/data/docker/volumes/runner-bb4211e4-project-60-concurrent-1-cache-3c3f060a0374fc8bc39395164f415a70/_data/awc500platform/oe-lite# ll build_*/cache.zip
-rw------- 1 root root 72292247293 Aug 16 15:56 build_ls1021a-1/cache.zip
-rw------- 1 root root  5709978495 Aug 23 14:14 build_ls1021a-2/cache.zip
-rw------- 1 root root 24039811662 Sep 16 16:49 build_ls1021a-5/cache.zip
-rw------- 1 root root 20905137413 Jan  3 14:19 build_ls1021a-6/cache.zip
-rw------- 1 root root  5834007514 Jan 20 13:46 build_ls1021a-7/cache.zip

I can see that the cache.zip files contains a lot of the same files.

Now my questions

Why do I see up to 7 cache.zip files in each volume?

  • build_ls1021a-1
  • build_ls1021a-2
  • etc

Why are there two volumes:

  • runner-bb4211e4-project-60-concurrent-0-cache-3c3f060a0374fc8bc39395164f415a70
  • runner-bb4211e4-project-60-concurrent-1-cache-3c3f060a0374fc8bc39395164f415a70

Could it be because the runner is allowed to run two jobs simultaneously?
But that would imply that -0 and -1 do not reuse caches.

Is it safe to enable some automatic deletion?

Thank You for any good suggestions

Our setup

Runner
We are running self managed
Runner version 13.10.0
GitLab Community Edition 14.3.4

.gitlab-ci.yml

.oelite:
  stage: build
  variables:
    GIT_SUBMODULE_STRATEGY: recursive
  tags:
    - wpt-bsp
  before_script:
    # Some configurations references deif.com, replace that if not running on gitlab.deif.com
    - sed -i "s/gitlab.deif.com/$CI_SERVER_HOST/mg" conf/*conf*
    # Fixup build and cache volume UID and GID
    - test "$HOME" = "/home/user" && sudo chown -R user:user . /home/user/*
  artifacts:
    paths:
      - images/
    expire_in: 2 weeks

build_ls1021a:
  extends: .oelite
  script:
    - ./scripts/ci-build.sh arm-ls1
    - rm -f images/*.md5
    - find images -type l -exec rm {} \;
  variables:
    TRANSFER_METER_FREQUENCY: 1s
    CACHE_COMPRESSION_LEVEL: fastest
  cache: 
    key: "$CI_JOB_NAME" 
    paths:
      - ingredients
      - tmp/packages

docker-compose.yml

version: '2'

services:

  runner:
    container_name: gitlab-runner
    image: gitlab/gitlab-runner:latest
    restart: always
    ports:
      - '10.0.0.84:8093:8093'
    volumes:
      - /srv/gitlab/runner/config:/etc/gitlab-runner
      - /var/run/docker.sock:/var/run/docker.sock

  cleanup:
    container_name: gitlab-runner-cleanup
    image: quay.io/gitlab/gitlab-runner-docker-cleanup:latest
    restart: always
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - /data:/data
    environment:
      - USE_DF=true
      - CHECK_PATH=/data
      - LOW_FREE_SPACE=50G
      - EXPECTED_FREE_SPACE=100G
      - LOW_FREE_FILES_COUNT=1048576
      - EXPECTED_FREE_FILES_COUNT=2097152
      - DEFAULT_TTL=10m

config.toml

[[runners]]
  name = "wptsrv2-bsp"
  limit = 2
  url = "https://gitlab.deif.com/"
  token = "***"
  executor = "docker"
  environment = ["FF_USE_FASTZIP=1", "INGREDIENTS=/ingredients"]
  [runners.cache]
  [runners.docker]
    tls_verify = false
    image = "gitlab.deif.com:5500/toolchains-docker/deif-base-debian"
    privileged = false
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/data/oelite/ingredients:/ingredients", "/cache"]
    pull_policy = ["always"]
    shm_size = 0