Hi all,
I have a Gitlab CE 14.10.5 self-host running on docker.
I have set Default artifacts expiration to 7 days and Clear the Keep the latest artifacts for all jobs in the latest successful pipelines checkbox
(They were 30 days and the Keep the latest artifacts for all jobs in the latest successful pipelines was checked before )
I added gitlab_rails['expire_build_artifacts_worker_cron'] = "*/10 * * * *" to /etc/gitlab/gitlab.rb
exec into gitlab container and cat /var/log/gitlab/sidekiq/current | grep ExpireBuildArtifactsWorker
{"severity":"INFO","time":"2023-09-22T05:50:33.225Z","retry":0,"queue":"cronjob:expire_build_artifacts","version":0,"queue_namespace":"cronjob","args":[],"class":"ExpireBuildArtifactsWorker","jid":"43c0290f62236f5085e07a15","created_at":"2023-09-22T05:50:33.216Z","meta.caller_id":"Cronjob","correlation_id":"9f74eda90e7a63be43835eff473ab3ae","meta.feature_category":"build_artifacts","worker_data_consistency":"always","idempotency_key":"resque:gitlab:duplicate:cronjob:expire_build_artifacts:7be9ea93da0bda6381b334cf2932692b41842a57ea0aeac11899369225b9bb7e","size_limiter":"validated","enqueued_at":"2023-09-22T05:50:33.217Z","job_size_bytes":2,"pid":32200,"message":"ExpireBuildArtifactsWorker JID-43c0290f62236f5085e07a15: start","job_status":"start","scheduling_latency_s":0.007936}
{"severity":"INFO","time":"2023-09-22T05:50:33.577Z","retry":3,"queue":"flush_counter_increments","version":0,"class":"FlushCounterIncrementsWorker","args":["[FILTERED]","59","[FILTERED]"],"at":1695362433.5755565,"jid":"a556e8643a52fd7f2062ba00","created_at":"2023-09-22T05:50:33.575Z","meta.caller_id":"ExpireBuildArtifactsWorker","correlation_id":"9f74eda90e7a63be43835eff473ab3ae","meta.feature_category":"build_artifacts","meta.client_id":"ip/","worker_data_consistency":"always","idempotency_key":"resque:gitlab:duplicate:flush_counter_increments:2b36124f4b376fd6d2928578c46b7eb8a5d96300c08101502c1bc723c6d1a49a","duplicate-of":"ae737a31bed79c23be1a2d91","job_size_bytes":47,"pid":32200,"job_status":"deduplicated","message":"FlushCounterIncrementsWorker JID-a556e8643a52fd7f2062ba00: deduplicated: dropped until executing","deduplication.type":"dropped until executing","deduplication.options.including_scheduled":true}
{"severity":"INFO","time":"2023-09-22T05:50:33.580Z","retry":3,"queue":"flush_counter_increments","version":0,"class":"FlushCounterIncrementsWorker","args":["[FILTERED]","59","[FILTERED]"],"at":1695362433.5793624,"jid":"c81292172b6c3786ae27fb18","created_at":"2023-09-22T05:50:33.579Z","meta.caller_id":"ExpireBuildArtifactsWorker","correlation_id":"9f74eda90e7a63be43835eff473ab3ae","meta.feature_category":"build_artifacts","meta.client_id":"ip/","worker_data_consistency":"always","idempotency_key":"resque:gitlab:duplicate:flush_counter_increments:93b90e28f57cf15d3058ff87f408c45803146515d15481a3718079da0aa2041a","duplicate-of":"5611818b604541c88cfd9bbf","job_size_bytes":39,"pid":32200,"job_status":"deduplicated","message":"FlushCounterIncrementsWorker JID-c81292172b6c3786ae27fb18: deduplicated: dropped until executing","deduplication.type":"dropped until executing","deduplication.options.including_scheduled":true}
{"severity":"INFO","time":"2023-09-22T05:50:35.177Z","retry":0,"queue":"cronjob:expire_build_artifacts","version":0,"queue_namespace":"cronjob","args":[],"class":"ExpireBuildArtifactsWorker","jid":"43c0290f62236f5085e07a15","created_at":"2023-09-22T05:50:33.216Z","meta.caller_id":"Cronjob","correlation_id":"9f74eda90e7a63be43835eff473ab3ae","meta.feature_category":"build_artifacts","worker_data_consistency":"always","idempotency_key":"resque:gitlab:duplicate:cronjob:expire_build_artifacts:7be9ea93da0bda6381b334cf2932692b41842a57ea0aeac11899369225b9bb7e","size_limiter":"validated","enqueued_at":"2023-09-22T05:50:33.217Z","job_size_bytes":2,"pid":32200,"message":"ExpireBuildArtifactsWorker JID-43c0290f62236f5085e07a15: done: 1.952091 sec","job_status":"done","scheduling_latency_s":0.007936,"redis_calls":7,"redis_duration_s":0.023356000000000002,"redis_read_bytes":106,"redis_write_bytes":1762,"redis_cache_calls":2,"redis_cache_duration_s":0.007204,"redis_cache_read_bytes":2,"redis_cache_write_bytes":348,"redis_queues_calls":1,"redis_queues_duration_s":0.001084,"redis_queues_read_bytes":86,"redis_queues_write_bytes":962,"redis_shared_state_calls":4,"redis_shared_state_duration_s":0.015068,"redis_shared_state_read_bytes":18,"redis_shared_state_write_bytes":452,"db_count":1009,"db_write_count":4,"db_cached_count":992,"db_replica_count":0,"db_primary_count":1009,"db_main_count":1009,"db_main_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":992,"db_main_cached_count":992,"db_main_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_main_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.457,"db_main_duration_s":0.457,"db_main_replica_duration_s":0.0,"cpu_s":1.399495,"mem_objects":712856,"mem_bytes":12163288,"mem_mallocs":113856,"mem_total_bytes":40677528,"extra.expire_build_artifacts_worker.destroyed_job_artifacts_count":2,"duration_s":1.952091,"completed_at":"2023-09-22T05:50:35.177Z","load_balancing_strategy":"primary","db_duration_s":0.50458}
gitlab-rake gitlab:cleanup:orphan_job_artifact_files DRY_RUN=false
I, [2023-09-22T07:36:11.949202 #134700] INFO -- : Processed 71857 job artifact(s) to find and cleaned 280 orphan(s).
It processed about 120k+ job artifacts and cleaned about 50k orphans first time a ran it.
It seems the cron job remove expired artifacts doesn’t work ?
I tried to delete all artifacts of projects via API
curl --request DELETE --header "PRIVATE-TOKEN: xxx" "https://gitlab.mydomain.com/api/v4/projects/305/artifacts"
{"message":"202 Accepted"}
==> worked
I can view from UI
ID: 305
Storage: 2.2 MB (Repository: 2.1 MB / Wikis: 0 Bytes / Build Artifacts: 0 Bytes / Pipeline Artifacts: 0 Bytes / LFS: 0 Bytes / Snippets: 0 Bytes / Packages: 0 Bytes / Uploads: 43 KB)
curl --request DELETE --header "PRIVATE-TOKEN: xxx" "https://gitlab.mydomain.com/api/v4/projects/112/artifacts"
{"message":"202 Accepted"}
==> not work
ID: 112
Storage: 5.2 GB (Repository: 331.6 MB / Wikis: 1.6 MB / Build Artifacts: 4.9 GB / Pipeline Artifacts: 0 Bytes / LFS: 0 Bytes / Snippets: 266 KB / Packages: 0 Bytes / Uploads: 0 Bytes)
My token is an admin gitlab token.
When I try to
curl --request DELETE --header "PRIVATE-TOKEN: xxx" "https://gitlab.mydomain.com/api/v4/projects/237/jobs/109861/artifacts"
But at UI I get this message
These artifacts are the latest. They will not be deleted (even if expired) until newer artifacts are available.
How can I troubleshoot it ? Please give me some advice , thank you very much.