Artifacts file system filling up since 14.7.1 upgrade

Since we upgraded to 14.7.1, our artifacts file system is filling up faster than before.

I ran several orphan cleanup tasks for artifacts, and they had no effect, despite listing > 8000 artifacts to be deleted during the first run (and yes, I did include DRY_RUN=false for the “real” run). Subsequent runs showed < 10 orphan artifacts to clean up. Since those runs, disk usage on the fs has increased from 70% to 87%, and it’s not a small fs. None of the runs showed any significant reduction in disk usage.

We have the “Keep the latest artifacts for all jobs in the latest successful pipelines” box unchecked instance-wide, since that bit us hard when it was introduced at the project level.

I have run orphan cleanups before (on earlier Gitlab releases), and they have reclaimed significant disk space, so this seems to be new.

I am aware of this issue: Clean up old expired job artifacts (#322817) · Issues · GitLab.org / GitLab · GitLab about cleaning up old expired artifacts, and we do have a lot of artifacts dating back to 2018-2019 that I would like to delete. I’m just not sure of the best way to do that before the fixes described in that issue are released.

If anyone can confirm that 14.7.2 or .3 will clean up old artifacts, we will go to it asap. If it’s safe to delete artifacts from 2018-2019 from the fs without db repercussions, we’ll do that. I’d just like reassurance from someone who is wiser than I.

Many thanks for any guidance!

Leslie

We have the same problem sind 14.7.1. Updated to .3 last Friday in hope it resolves the problem. But didn’t. We’re deleting the expired artifacts manually via a cron job until Gitlab fixes this.

We noticed this problem on Feb 18 (GitLab 14.7.2)

Cleanup doesn’t help:

gitlab-rake gitlab:cleanup:orphan_job_artifact_files NICENESS=Idle DRY_RUN=false

UI says that artifacts were removed, and download button is hidden. But in fact no artifacts are removed, they are still exists on the disk, and can be downloaded using direct link.

Screenshot 2022-02-23 at 13.32.28550×660 90.8 KB

The standard cleanup job doesn’t work for us either.
We composed the following script and run it via the rails console twice a day.

builds_with_artifacts = Ci::Build.with_downloadable_artifacts
builds_to_clear = builds_with_artifacts.where("artifacts_expire_at < ?", Time.now)
builds_to_clear.find_each do |build|
  build.erase_erasable_artifacts!
end

rkeck, you are my hero!!!

I will try this tonight.

Thanks for reporting. Suggest the following issues to follow:

Clean up old expired job artifacts on gitlab.com (#322817) · Issues · GitLab.org / GitLab · GitLab - for engineering updates, GitLab.com SaaS.
Expired artifacts still taking disk space (#346261) · Issues · GitLab.org / GitLab · GitLab - for adding your reports, and discussing workarounds and ideas for self-managed instances.

Problem is fixed with 14.8.4 BTW

We have upgraded to 14.8.4, but it seems the features are still not enabled there.

Is that your experience? Have you enabled either of them? (ci_destroy_all_expired_service, ci_detect_wrongly_expired_artifacts)