Repo Storage keeps growing despite jobs expiring

GitLab CI/CD
1

Hello,

I’m using gitlab.com with shared runners and i have a 27MB Files repo with some CI/CD pipelines, the thing is that now my Storage Size is 7.4GB and it’s increasing very fast.

I checked the stats of the repo through the api and got this :

“statistics”: {
“commit_count”: 648,
“storage_size”: 7897162278,
“repository_size”: 28353495,
“wiki_size”: 0,
“lfs_objects_size”: 0,
“job_artifacts_size”: 7868808783,
“snippets_size”: 0
},

so i proceeded to delete all the jobs from the api, but as expected it reported to me that there were 0 jobs so, instead, i deleted almost all the pipelines. From more than 200 pipelines i now have only 10, but the storage size is still 7.4gb and the stats keeps reporting the same, “job_artifacts_size”: 7868808783.

I don’t know what else to try, how i’m supposed to free that job artifacts size reported by statistics if the api says that i have 0 jobs?

Can someone guide me in the right direction? Thanks

2

This is not a direct answer on how to cleanup on gitlab, as I see only jobs to run from commandline (see https://docs.gitlab.com/ee/administration/job_artifacts.html#job-artifacts-using-too-much-disk-space ), but maybe you can do something with this.

Maybe you can do a cleanup via the API, but I haven’'t tested that yet: https://docs.gitlab.com/ee/api/job_artifacts.html

For the future, you can expire artifacts sooner to have less stored: https://docs.gitlab.com/ee/administration/job_artifacts.html#expiring-artifacts

3

Thanks, but i already tried to do a cleanup via the API with no luck, i guess i did something wrong and after deleting the pipelines i can’t retrieve the job ids to call an /erase on the job.

The thing is i deleted all the pipelines after getting 0 jobs from the url /project_id/jobs and now i can’t get the ids if they keep existing

4

Hello,
from my experience on our self-hosted gitlab :

  • expiring artifacts works, but it won’t expire the logs (even though those are stored and exposed in the API as artifacts)
  • erasing the jobs, then deleting the pipeline works (I have a script for that)
  • because of this bug https://gitlab.com/gitlab-org/gitlab/-/issues/224151, deleting the pipeline without first erasing its jobs creates dandling job artifacts on disc. Admins can delete the dandling artifacts, but I don’t know how to fix the quotas afterwards.
5

Hm, in the same forum update email I see gitlab uses PostgreSQL13.3. Seems there needs to be some RI constraints between pipelines and jobs!