Summary
As a GitLab SaaS customer, we noticed that our reported usage kept on going up even though all artifacts are expired.
The web UI correctly reports these artifacts as being removed (since they indeed expired), but the API reports these job artifacts.zip to still exist.
Finding and deleting these pipelines corrects the situation caused by this bug.
Details
Pipelines of our single (private) project generate pretty large artifacts (~2GB). However all artifacts are set to expire quickly (expire_in: 1 hour or expire_in: 2 hours for one particular job) since we upload them elsewhere for long-term storage.
Nonetheless, today we almost hit or maximum usage quota (29.4 GB of 30 GB, see screenshot below) even though all artifacts should have been expired. We already purchased two additional storage packs (+10GB each) to raise our available usage quota to 30GB.
This might be related to epic &5380 Statistics artifacts_size is broken, not sure.
Analysis
To find the culprits, we wrote a script to query the GitLab API and report all job artifact sizes of all pipelines and found 10 pipelines still having job artifacts although the web UI reports them as already long been deleted.
After manually deleting these pipelines our usage went down again (had to wait a couple of minutes for it to update). We left one untouched, see “Example” below.
Pipeline #415888242: 2441 MB <-- left untouched (see "Example" below)
Pipeline #415050560: 1193 MB
Pipeline #413799439: 2089 MB
Pipeline #401832642: 1244 MB
Pipeline #401467520: 2457 MB
Pipeline #401439793: 2062 MB
Pipeline #400920092: 1018 MB
Pipeline #400623556: 2058 MB
Pipeline #393868228: 2040 MB
Pipeline #393733871: 2099 MB
Example
For analysis, we left one of these occurrences untouched for now, in case a GitLab staff member wants to take a look at this bug, i.e. pipeline #415888242 for which the API reports jobs still having artifacts.zip (totaling 2441 MB) but not the web UI (see screenshot below).
API
| Pipeline | Job | Artifact | Size |
|-----------|------------|---------------|-----------|
| 415888242 | 1817496744 | job.log | 11320 |
| 415888242 | 1817496739 | job.log | 4385 |
| 415888242 | 1817496733 | job.log | 2862 |
| 415888242 | 1817496731 | job.log | 2842 |
| 415888242 | 1817496725 | artifacts.zip | 638777710 |
| 415888242 | 1817496725 | metadata.gz | 224 |
| 415888242 | 1817496725 | job.log | 122487 |
| 415888242 | 1817496718 | artifacts.zip | 623539671 |
| 415888242 | 1817496718 | metadata.gz | 219 |
| 415888242 | 1817496718 | job.log | 55452 |
| 415888242 | 1817496699 | artifacts.zip | 654635452 |
| 415888242 | 1817496699 | metadata.gz | 34213 |
| 415888242 | 1817496699 | job.log | 2035722 |
| 415888242 | 1817496661 | artifacts.zip | 638465754 |
| 415888242 | 1817496661 | metadata.gz | 16090 |
| 415888242 | 1817496661 | job.log | 1809718 |
| 415888242 | 1817496649 | artifacts.zip | 150361 |
| 415888242 | 1817496649 | metadata.gz | 243 |
| 415888242 | 1817496649 | job.log | 59759 |
| 415888242 | 1817496640 | artifacts.zip | 137892 |
| 415888242 | 1817496640 | metadata.gz | 243 |
| 415888242 | 1817496640 | job.log | 59936 |
| 415888242 | 1817496632 | job.log | 4707 |
| 415888242 | 1817496632 | .env.gz | 176 |
Web
For those six jobs of the pipeline having artifacts.zip above, the web UI shows them as already been deleted:
Purchasing more additional storage packs is not a sustainable solution 