I am currently running Gitlab CE 16.6.1 self-managed with Docker, and Gitlab Runner latest within the same machine.
I noticed that every projects taking 3 docker volumes when its pipeline runs. Since we have so many Projects and quiet active deployment, docker volumes takes so many storage in my server.
I always manually prune this volume at weekly basis, since it is only Pipeline Cache (I guess)…
Is it really safe to manually Prune these docker volumes?
How can I control this behavior, maybe like:
Limiting docker volume for pipeline runs
Docker volume pruning schedule
etc
root@gitlab-server:~# docker volume ls
local runner-zq1mkdd7-project-264-concurrent-0-cache-3c3f060a0374fc8bc39395164f415a70
local runner-zq1mkdd7-project-264-concurrent-0-cache-904f6ed42e0fa2b14c1d7a2ed6f1875e
local runner-zq1mkdd7-project-264-concurrent-0-cache-c33bcaa1fd2c77edfc3893b41966cea8
local runner-zq1mkdd7-project-271-concurrent-0-cache-3c3f060a0374fc8bc39395164f415a70
local runner-zq1mkdd7-project-271-concurrent-0-cache-904f6ed42e0fa2b14c1d7a2ed6f1875e
local runner-zq1mkdd7-project-271-concurrent-0-cache-c33bcaa1fd2c77edfc3893b41966cea8
local runner-zq1mkdd7-project-273-concurrent-0-cache-3c3f060a0374fc8bc39395164f415a70
local runner-zq1mkdd7-project-273-concurrent-0-cache-904f6ed42e0fa2b14c1d7a2ed6f1875e
local runner-zq1mkdd7-project-273-concurrent-0-cache-c33bcaa1fd2c77edfc3893b41966cea8
local runner-zq1mkdd7-project-282-concurrent-0-cache-3c3f060a0374fc8bc39395164f415a70
local runner-zq1mkdd7-project-282-concurrent-0-cache-904f6ed42e0fa2b14c1d7a2ed6f1875e
local runner-zq1mkdd7-project-282-concurrent-0-cache-c33bcaa1fd2c77edfc3893b41966cea8
local runner-zq1mkdd7-project-302-concurrent-0-cache-3c3f060a0374fc8bc39395164f415a70
local runner-zq1mkdd7-project-302-concurrent-0-cache-904f6ed42e0fa2b14c1d7a2ed6f1875e
local runner-zq1mkdd7-project-302-concurrent-0-cache-c33bcaa1fd2c77edfc3893b41966cea8
local runner-zq1mkdd7-project-314-concurrent-0-cache-3c3f060a0374fc8bc39395164f415a70
local runner-zq1mkdd7-project-314-concurrent-0-cache-904f6ed42e0fa2b14c1d7a2ed6f1875e
local runner-zq1mkdd7-project-314-concurrent-0-cache-c33bcaa1fd2c77edfc3893b41966cea8
local runner-zq1mkdd7-project-331-concurrent-0-cache-3c3f060a0374fc8bc39395164f415a70
local runner-zq1mkdd7-project-331-concurrent-0-cache-904f6ed42e0fa2b14c1d7a2ed6f1875e
local runner-zq1mkdd7-project-331-concurrent-0-cache-c33bcaa1fd2c77edfc3893b41966cea8
local runner-zq1mkdd7-project-345-concurrent-0-cache-3c3f060a0374fc8bc39395164f415a70
local runner-zq1mkdd7-project-345-concurrent-0-cache-904f6ed42e0fa2b14c1d7a2ed6f1875e
local runner-zq1mkdd7-project-345-concurrent-0-cache-c33bcaa1fd2c77edfc3893b41966cea8
local runner-zq1mkdd7-project-347-concurrent-0-cache-3c3f060a0374fc8bc39395164f415a70
local runner-zq1mkdd7-project-347-concurrent-0-cache-904f6ed42e0fa2b14c1d7a2ed6f1875e
local runner-zq1mkdd7-project-347-concurrent-0-cache-c33bcaa1fd2c77edfc3893b41966cea8
local runner-zq1mkdd7-project-373-concurrent-0-cache-3c3f060a0374fc8bc39395164f415a70
local runner-zq1mkdd7-project-373-concurrent-0-cache-904f6ed42e0fa2b14c1d7a2ed6f1875e
local runner-zq1mkdd7-project-373-concurrent-0-cache-c33bcaa1fd2c77edfc3893b41966cea8
local runner-zq1mkdd7-project-419-concurrent-0-cache-3c3f060a0374fc8bc39395164f415a70
local runner-zq1mkdd7-project-419-concurrent-0-cache-904f6ed42e0fa2b14c1d7a2ed6f1875e
local runner-zq1mkdd7-project-419-concurrent-0-cache-c33bcaa1fd2c77edfc3893b41966cea8
...
Yes, it’s just the project cache, so it’s safe to delete
AFAIK, there does not exist any automatic cache cleanup by GitLab, therefore you will have to implement something by yourself. Apparently, there exists a script (I believe it’s just docker volume prune with some added logic to it), but I personally never tried it. You could maybe use this to make a cron job on the server and clean it up this way 1x a week.
P.S. Normally this gets automatically cleaned whenever you update the runner (that’s how I clean it xD)
Yes, I have read the Documentations about Docker Cache as well and found that custom scripts inside my gitlab-runner container > /usr/share/gitlab-runner folder. But I am not so sure how to run the scripts within my Docker scenarios.
P.S. Normally this gets automatically cleaned whenever you update the runner (that’s how I clean it xD)
Can you explain more about “update the runner” ??? I always use gitlab-runner:latest image
AFAIK, you need to manually update runner if you want - even if you use latest, docker does not pull it every day - have a look at: Run GitLab Runner in a container | GitLab
My Runners are installed directly on Linux (not in docker), so for me this works fine. Not sure how cache is controlled with runners in docker.
In your case, I guess you could make a cron job that enters the container and executes that script… so something like docker exec <gitlab-runner-container-name> <cleanup-script>. But again, I’ve never done this, but if I would, this is probably the way I’d try.
Just a quick query on this as our GitLab Runner containers continually eat all available space, and one cannot run /usr/share/gitlab-runner/clear-docker-cache from within the container.
Is it truly safe to delete these runner-CODE-project-NUMBER... volumes?
When I have tried this in the past I have started to receive Pipeline errors like:
Using Docker executor with image cypress/browsers:node-18.14.1-chrome-110.0.5481.96-1-ff-109.0-edge-110.0.1587.41-1 ...
ERROR: Preparation failed: adding cache volume: set volume permissions: running permission container "43ba36256b354cb28b0c24a097792f2d6e9ce2912e8a08c1a7f4d1b03f84d162" for volume "runner-xskszy3f-project-1502-concurrent-0-cache-3c3f060a0374fc8bc39395164f415a70": starting permission container: Error response from daemon: error evaluating symlinks from mount source "/var/lib/docker/volumes/runner-xskszy3f-project-1502-concurrent-0-cache-3c3f060a0374fc8bc39395164f415a70/_data": lstat /var/lib/docker/volumes/runner-xskszy3f-project-1502-concurrent-0-cache-3c3f060a0374fc8bc39395164f415a70: no such file or directory (linux_set.go:105:0s)
Note the no such file or directory at the end.
No amount of clicking on “Clear runner caches”, restarts, nor anything else resolved it. The only solution was to totally destory the Runner and go through the set-up process again.
I really do not want to have to spend my time continually destroying and rebuilding Runners.
How is your Runner installed? Which executor are you using? What is the OS?
I’m mostly using Runners installed with official Linux repositories, on Ubuntu, docker executor. I’ve cleaned those volumes like millions of times (docker volume prune --all when Runner is paused to make sure it is not executing any job) and never had such an issue. I have an issue with docker executor on Windows Server, but that’s another story and even has an issue open: gitlab-runner on Windows Server thorws "Access denied" errors (#28917) · Issues · GitLab.org / gitlab-runner · GitLab
Hosts are a mix of CentOS 7 and Ubuntu 22.04. All the runners are deployed as Docker containers and all are using the Docker executor.
The runner volumes are building up and given what happened to me last time, there does not appear to be a safe way to delete them given that “Clear runner caches” doesn’t seem to do anything.
