Artifacts upload broken after upgrade to 15.0.0

I’m not completely sure how to debug this one. Runner is trying to upload artifacts after successful job and fails getting 400 bad request:

Uploading artifacts for successful job
00:09
Uploading artifacts...
klepto*.pkg.tar.zst: found 1 matching files and directories 
WARNING: Uploading artifacts as "archive" to coordinator... 400 Bad Request  id=12730 responseStatus=400 Bad Request status=400 token=eHWKDVMU
WARNING: Retrying...                                context=artifacts-uploader error=invalid argument
WARNING: Uploading artifacts as "archive" to coordinator... 400 Bad Request  id=12730 responseStatus=400 Bad Request status=400 token=eHWKDVMU
WARNING: Retrying...                                context=artifacts-uploader error=invalid argument
WARNING: Uploading artifacts as "archive" to coordinator... 400 Bad Request  id=12730 responseStatus=400 Bad Request status=400 token=eHWKDVMU
FATAL: invalid argument  

I can see the error in workhorse.log:

127.0.0.1 127.0.0.1 - - [2022/05/29:10:09:05 +0200] "POST /api/v4/jobs/12730/artifacts?artifact_format=zip&artifact_type=archive&expire_in=1+day HTTP/1.1" 400 47 "" "gitlab-runner 15.0.0 (15-0-stable; go1.17.7; linux/amd64)" 1802
127.0.0.1 2001:4ba0:fff1:c3::1 - - [2022/05/29:10:09:06 +0200] "POST /api/v4/jobs/request HTTP/1.1" 204 0 "" "gitlab-runner 15.0.0 (15-0-stable; go1.17.7; linux/amd64)" 27
127.0.0.1 2003:cb:873c:bb54:6398:fa51:4edd:d3f0 - - [2022/05/29:10:09:07 +0200] "GET /arch/klepto/-/jobs/12730/trace.json?state=eyJvZmZzZXQiOjExMDExLCJzdHlsZSI6eyJmZyI6bnVsbCwiYmciOm51bGwsIm1hc2siOjB9LCJvcGVuX3NlY3Rpb25zIjp7InN0ZXAtc2NyaXB0IjoiMTY1MzgxMTY3MSJ9fQ%3D%3D HTTP/1.1" 200 2435 "https://git.jan-kohnert.de/arch/klepto/-/jobs/12730" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.64 Safari/537.36" 99
127.0.0.1 2001:4ba0:fff1:c3::1 - - [2022/05/29:10:09:07 +0200] "PATCH /api/v4/jobs/12730/trace HTTP/1.1" 202 1 "" "gitlab-runner 15.0.0 (15-0-stable; go1.17.7; linux/amd64)" 48
time="2022-05-29T10:09:08+02:00" level=info msg="saved file" client_mode=local_tempfile copied_bytes=3910285 correlation_id=01G47DS5W4FEAY6Q6SRX2R957F is_local=true is_multipart=false is_remote=false local_temp_path=/opt/gitlab/gitlab/shared/artifacts/tmp/uploads remote_id= temp_file_prefix=build-klepto-master.zip
time="2022-05-29T10:09:08+02:00" level=info msg="saved file" client_mode=local_tempfile copied_bytes=179 correlation_id=01G47DS5W4FEAY6Q6SRX2R957F is_local=true is_multipart=false is_remote=false local_temp_path=/tmp remote_id= temp_file_prefix=metadata.gz
127.0.0.1 127.0.0.1 - - [2022/05/29:10:09:08 +0200] "POST /api/v4/jobs/12730/artifacts?artifact_format=zip&artifact_type=archive&expire_in=1+day HTTP/1.1" 400 47 "" "gitlab-runner 15.0.0 (15-0-stable; go1.17.7; linux/amd64)" 1383
127.0.0.1 2001:4ba0:fff1:c3::1 - - [2022/05/29:10:09:09 +0200] "POST /api/v4/jobs/request HTTP/1.1" 204 0 "" "gitlab-runner 15.0.0 (15-0-stable; go1.17.7; linux/amd64)" 29
127.0.0.1 2001:4ba0:fff1:c3::1 - - [2022/05/29:10:09:10 +0200] "PATCH /api/v4/jobs/12730/trace HTTP/1.1" 202 1 "" "gitlab-runner 15.0.0 (15-0-stable; go1.17.7; linux/amd64)" 54
127.0.0.1 2003:cb:873c:bb54:6398:fa51:4edd:d3f0 - - [2022/05/29:10:09:11 +0200] "GET /arch/klepto/-/jobs/12730/trace.json?state=eyJvZmZzZXQiOjExODIxLCJzdHlsZSI6eyJmZyI6bnVsbCwiYmciOm51bGwsIm1hc2siOjB9LCJvcGVuX3NlY3Rpb25zIjp7InVwbG9hZC1hcnRpZmFjdHMtb24tc3VjY2VzcyI6IjE2NTM4MTE3NDMifX0%3D HTTP/1.1" 200 1847 "https://git.jan-kohnert.de/arch/klepto/-/jobs/12730" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.64 Safari/537.36" 93
time="2022-05-29T10:09:11+02:00" level=info msg="saved file" client_mode=local_tempfile copied_bytes=3910285 correlation_id=01G47DS9C9773HXCS5D0S47N6Z is_local=true is_multipart=false is_remote=false local_temp_path=/opt/gitlab/gitlab/shared/artifacts/tmp/uploads remote_id= temp_file_prefix=build-klepto-master.zip
time="2022-05-29T10:09:11+02:00" level=info msg="saved file" client_mode=local_tempfile copied_bytes=179 correlation_id=01G47DS9C9773HXCS5D0S47N6Z is_local=true is_multipart=false is_remote=false local_temp_path=/tmp remote_id= temp_file_prefix=metadata.gz
127.0.0.1 127.0.0.1 - - [2022/05/29:10:09:11 +0200] "POST /api/v4/jobs/12730/artifacts?artifact_format=zip&artifact_type=archive&expire_in=1+day HTTP/1.1" 400 47 "" "gitlab-runner 15.0.0 (15-0-stable; go1.17.7; linux/amd64)" 1502
127.0.0.1 2001:4ba0:fff1:c3::1 - - [2022/05/29:10:09:12 +0200] "POST /api/v4/jobs/request HTTP/1.1" 204 0 "" "gitlab-runner 15.0.0 (15-0-stable; go1.17.7; linux/amd64)" 19
127.0.0.1 2003:cb:873c:bb54:6398:fa51:4edd:d3f0 - - [2022/05/29:10:09:12 +0200] "GET /arch/klepto/-/jobs/12730.json HTTP/1.1" 200 5921 "https://git.jan-kohnert.de/arch/klepto/-/jobs/12730" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.64 Safari/537.36" 181
127.0.0.1 2001:4ba0:fff1:c3::1 - - [2022/05/29:10:09:12 +0200] "PATCH /api/v4/jobs/12730/trace HTTP/1.1" 202 1 "" "gitlab-runner 15.0.0 (15-0-stable; go1.17.7; linux/amd64)" 49
127.0.0.1 2001:4ba0:fff1:c3::1 - - [2022/05/29:10:09:12 +0200] "PUT /api/v4/jobs/12730 HTTP/1.1" 200 3 "" "gitlab-runner 15.0.0 (15-0-stable; go1.17.7; linux/amd64)" 128
127.0.0.1 2001:4ba0:fff1:c3::1 - - [2022/05/29:10:09:15 +0200] "POST /api/v4/jobs/request HTTP/1.1" 204 0 "" "gitlab-runner 15.0.0 (15-0-stable; go1.17.7; linux/amd64)" 27
127.0.0.1 2003:cb:873c:bb54:6398:fa51:4edd:d3f0 - - [2022/05/29:10:09:15 +0200] "GET /arch/klepto/-/jobs/12730/trace.json?state=eyJvZmZzZXQiOjEyNTAzLCJzdHlsZSI6eyJmZyI6bnVsbCwiYmciOm51bGwsIm1hc2siOjB9LCJvcGVuX3NlY3Rpb25zIjp7InVwbG9hZC1hcnRpZmFjdHMtb24tc3VjY2VzcyI6IjE2NTM4MTE3NDMifX0%3D HTTP/1.1" 200 1380 "https://git.jan-kohnert.de/arch/klepto/-/jobs/12730" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.64 Safari/537.36" 73
127.0.0.1 2001:4ba0:fff1:c3::1 - - [2022/05/29:10:09:18 +0200] "POST /api/v4/jobs/request HTTP/1.1" 204 0 "" "gitlab-runner 15.0.0 (15-0-stable; go1.17.7; linux/amd64)" 25
127.0.0.1 2001:4ba0:fff1:c3::1 - - [2022/05/29:10:09:21 +0200] "POST /api/v4/jobs/request HTTP/1.1" 204 0 "" "gitlab-runner 15.0.0 (15-0-stable; go1.17.7; linux/amd64)" 72
127.0.0.1 2003:cb:873c:bb54:6398:fa51:4edd:d3f0 - - [2022/05/29:10:09:22 +0200] "GET /arch/klepto/-/jobs/12730.json HTTP/1.1" 200 7369 "https://git.jan-kohnert.de/arch/klepto/-/jobs/12730" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.64 Safari/537.36" 238
127.0.0.1 2001:4ba0:fff1:c3::1 - - [2022/05/29:10:09:24 +0200] "POST /api/v4/jobs/request HTTP/1.1" 204 0 "" "gitlab-runner 15.0.0 (15-0-stable; go1.17.7; linux/amd64)" 24

However, trying to narrow down the reason does in production.log not really give me a clue:

Started POST "/api/v4/jobs/12730/artifacts/authorize?artifact_format=zip&artifact_type=archive&expire_in=1+day" for 10.10.0.56 at 2022-05-29 10:09:03 +0200
Started PATCH "/api/v4/jobs/12730/trace" for 2001:4ba0:fff1:c3::1 at 2022-05-29 10:09:04 +0200
Started POST "/api/v4/jobs/12730/artifacts?artifact_format=zip&artifact_type=archive&expire_in=1+day" for 10.10.0.56 at 2022-05-29 10:09:05 +0200
Started POST "/api/v4/jobs/request" for 2001:4ba0:fff1:c3::1 at 2022-05-29 10:09:06 +0200
Started POST "/api/v4/jobs/12730/artifacts/authorize?artifact_format=zip&artifact_type=archive&expire_in=1+day" for 10.10.0.56 at 2022-05-29 10:09:06 +0200
Started GET "/arch/klepto/-/jobs/12730/trace.json?state=eyJvZmZzZXQiOjExMDExLCJzdHlsZSI6eyJmZyI6bnVsbCwiYmciOm51bGwsIm1hc2siOjB9LCJvcGVuX3NlY3Rpb25zIjp7InN0ZXAtc2NyaXB0IjoiMTY1MzgxMTY3MSJ9fQ%3D%3D" for 2003:cb:873c:bb54:6398:fa51:4edd:d3f0 at 2022-05-29 10:09:07 +0200
Processing by Projects::JobsController#trace as JSON
  Parameters: {"state"=>"eyJvZmZzZXQiOjExMDExLCJzdHlsZSI6eyJmZyI6bnVsbCwiYmciOm51bGwsIm1hc2siOjB9LCJvcGVuX3NlY3Rpb25zIjp7InN0ZXAtc2NyaXB0IjoiMTY1MzgxMTY3MSJ9fQ==", "namespace_id"=>"arch", "project_id"=>"klepto", "id"=>"12730"}
Completed 200 OK in 38ms (Views: 1.7ms | ActiveRecord: 3.8ms | Elasticsearch: 0.0ms | Allocations: 9526)
Started PATCH "/api/v4/jobs/12730/trace" for 2001:4ba0:fff1:c3::1 at 2022-05-29 10:09:07 +0200
Started POST "/api/v4/jobs/12730/artifacts?artifact_format=zip&artifact_type=archive&expire_in=1+day" for 10.10.0.56 at 2022-05-29 10:09:08 +0200
Started POST "/api/v4/jobs/request" for 2001:4ba0:fff1:c3::1 at 2022-05-29 10:09:09 +0200
Started POST "/api/v4/jobs/12730/artifacts/authorize?artifact_format=zip&artifact_type=archive&expire_in=1+day" for 10.10.0.56 at 2022-05-29 10:09:10 +0200
Started PATCH "/api/v4/jobs/12730/trace" for 2001:4ba0:fff1:c3::1 at 2022-05-29 10:09:10 +0200
Started GET "/arch/klepto/-/jobs/12730/trace.json?state=eyJvZmZzZXQiOjExODIxLCJzdHlsZSI6eyJmZyI6bnVsbCwiYmciOm51bGwsIm1hc2siOjB9LCJvcGVuX3NlY3Rpb25zIjp7InVwbG9hZC1hcnRpZmFjdHMtb24tc3VjY2VzcyI6IjE2NTM4MTE3NDMifX0%3D" for 2003:cb:873c:bb54:6398:fa51:4edd:d3f0 at 2022-05-29 10:09:11 +0200
Processing by Projects::JobsController#trace as JSON
  Parameters: {"state"=>"eyJvZmZzZXQiOjExODIxLCJzdHlsZSI6eyJmZyI6bnVsbCwiYmciOm51bGwsIm1hc2siOjB9LCJvcGVuX3NlY3Rpb25zIjp7InVwbG9hZC1hcnRpZmFjdHMtb24tc3VjY2VzcyI6IjE2NTM4MTE3NDMifX0=", "namespace_id"=>"arch", "project_id"=>"klepto", "id"=>"12730"}
Completed 200 OK in 39ms (Views: 1.9ms | ActiveRecord: 3.7ms | Elasticsearch: 0.0ms | Allocations: 9739)
Started POST "/api/v4/jobs/12730/artifacts?artifact_format=zip&artifact_type=archive&expire_in=1+day" for 10.10.0.56 at 2022-05-29 10:09:11 +0200
Started POST "/api/v4/jobs/request" for 2001:4ba0:fff1:c3::1 at 2022-05-29 10:09:12 +0200
Started GET "/api/v4/geo/proxy" for 127.0.0.1 at 2022-05-29 10:09:12 +0200
Started GET "/arch/klepto/-/jobs/12730.json" for 2003:cb:873c:bb54:6398:fa51:4edd:d3f0 at 2022-05-29 10:09:12 +0200
Processing by Projects::JobsController#show as JSON
  Parameters: {"namespace_id"=>"arch", "project_id"=>"klepto", "id"=>"12730"}
Completed 200 OK in 153ms (Views: 2.4ms | ActiveRecord: 26.2ms | Elasticsearch: 0.0ms | Allocations: 37536)
Started PATCH "/api/v4/jobs/12730/trace" for 2001:4ba0:fff1:c3::1 at 2022-05-29 10:09:12 +0200
Started PUT "/api/v4/jobs/12730" for 2001:4ba0:fff1:c3::1 at 2022-05-29 10:09:12 +0200
  Rendered layout layouts/mailer.html.haml (Duration: 95.3ms | Allocations: 19213)
  Rendered layout layouts/mailer.text.erb (Duration: 11.1ms | Allocations: 2003)
Delivered mail 62932a2982320_30ae494d026349@b079.mail (67.3ms)

Also api_json.log does not really reveal any more hints:

{"time":"2022-05-29T08:09:03.820Z","severity":"INFO","duration_s":0.02315,"db_duration_s":0.00468,"view_duration_s":0.01847,"status":200,"method":"POST","path":"/api/v4/jobs/12730/artifacts/authorize","params":[{"key":"artifact_format","value":"zip"},{"key":"artifact_type","value":"archive"},{"key":"expire_in","value":"1 day"}],"host":"git.jan-kohnert.de","remote_ip":"127.0.0.1, 10.10.0.56","ua":"gitlab-runner 15.0.0 (15-0-stable; go1.17.7; linux/amd64)","route":"/api/:version/jobs/:id/artifacts/authorize","redis_calls":5,"redis_duration_s":0.001288,"redis_read_bytes":405,"redis_write_bytes":293,"redis_cache_calls":2,"redis_cache_duration_s":0.00055,"redis_cache_read_bytes":405,"redis_cache_write_bytes":119,"redis_shared_state_calls":3,"redis_shared_state_duration_s":0.000738,"redis_shared_state_write_bytes":174,"db_count":6,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":6,"db_main_count":6,"db_main_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_main_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_main_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.005,"db_main_duration_s":0.005,"db_main_replica_duration_s":0.0,"cpu_s":0.032053,"pid":28215,"rate_limiting_gates":[],"correlation_id":"01G47DS30Y3WVG62Y2YPP8H58Y","meta.user":"jan","meta.project":"arch/klepto","meta.root_namespace":"arch","meta.client_id":"ip/10.10.0.56","meta.caller_id":"POST /api/:version/jobs/:id/artifacts/authorize","meta.remote_ip":"10.10.0.56","meta.feature_category":"build_artifacts","meta.pipeline_id":3057,"meta.job_id":12730,"content_length":"0","request_urgency":"low","target_duration_s":5}
{"time":"2022-05-29T08:09:04.541Z","severity":"INFO","duration_s":0.02196,"db_duration_s":0.00575,"view_duration_s":0.01621,"status":202,"method":"PATCH","path":"/api/v4/jobs/12730/trace","params":[],"host":"git.jan-kohnert.de","remote_ip":"127.0.0.1, 2001:4ba0:fff1:c3::1, 2001:4ba0:fff1:c3::1","ua":"gitlab-runner 15.0.0 (15-0-stable; go1.17.7; linux/amd64)","route":"/api/:version/jobs/:id/trace","queue_duration_s":0.010665,"redis_calls":7,"redis_duration_s":0.001875,"redis_read_bytes":5,"redis_write_bytes":633,"redis_cache_calls":1,"redis_cache_duration_s":0.000248,"redis_cache_read_bytes":2,"redis_cache_write_bytes":144,"redis_shared_state_calls":6,"redis_shared_state_duration_s":0.001627,"redis_shared_state_read_bytes":3,"redis_shared_state_write_bytes":489,"db_count":7,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":7,"db_main_count":7,"db_main_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_main_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_main_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.006,"db_main_duration_s":0.006,"db_main_replica_duration_s":0.0,"cpu_s":0.027003,"pid":28156,"rate_limiting_gates":[],"correlation_id":"01G47DS3QPMHK6YKZZR5FAJYQH","meta.user":"jan","meta.project":"arch/klepto","meta.root_namespace":"arch","meta.client_id":"ip/2001:4ba0:fff1:c3::1","meta.caller_id":"PATCH /api/:version/jobs/:id/trace","meta.remote_ip":"2001:4ba0:fff1:c3::1","meta.feature_category":"continuous_integration","meta.pipeline_id":3057,"meta.job_id":12730,"content_length":"810","content_range":"11011-11820","request_urgency":"low","target_duration_s":5}
{"time":"2022-05-29T08:09:06.160Z","severity":"INFO","duration_s":0.01236,"db_duration_s":0.00236,"view_duration_s":0.01,"status":204,"method":"POST","path":"/api/v4/jobs/request","params":[{"key":"info","value":{"name":"gitlab-runner","version":"15.0.0","revision":"febb2a09","platform":"linux","architecture":"amd64","executor":"kubernetes","shell":"bash","features":{"variables":"[FILTERED]","image":null,"services":null,"artifacts":null,"cache":null,"shared":null,"upload_multiple_artifacts":null,"upload_raw_artifacts":null,"session":null,"terminal":null,"refspecs":null,"masking":null,"proxy":null,"raw_variables":"[FILTERED]","artifacts_exclude":null,"multi_build_steps":null,"trace_reset":"[FILTERED]","trace_checksum":"[FILTERED]","trace_size":"[FILTERED]","vault_secrets":"[FILTERED]","cancelable":null,"return_exit_code":null,"service_variables":"[FILTERED]"},"config":{"gpus":""}}},{"key":"token","value":"[FILTERED]"},{"key":"last_update","value":"ea81f7047ca3378fd10e27b9efbaa4f7"}],"host":"git.jan-kohnert.de","remote_ip":"127.0.0.1, 2001:4ba0:fff1:c3::1, 2001:4ba0:fff1:c3::1","ua":"gitlab-runner 15.0.0 (15-0-stable; go1.17.7; linux/amd64)","route":"/api/:version/jobs/request","queue_duration_s":0.012716,"redis_calls":7,"redis_duration_s":0.0025729999999999998,"redis_read_bytes":34,"redis_write_bytes":662,"redis_cache_calls":1,"redis_cache_duration_s":0.000322,"redis_cache_read_bytes":2,"redis_cache_write_bytes":242,"redis_shared_state_calls":6,"redis_shared_state_duration_s":0.002251,"redis_shared_state_read_bytes":32,"redis_shared_state_write_bytes":420,"db_count":2,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":2,"db_main_count":2,"db_main_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_main_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_main_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.002,"db_main_duration_s":0.002,"db_main_replica_duration_s":0.0,"cpu_s":0.021055,"pid":28156,"rate_limiting_gates":[],"correlation_id":"01G47DS5APX79K0NCDDNYS9BG5","meta.client_id":"runner/7","meta.caller_id":"POST /api/:version/jobs/request","meta.remote_ip":"2001:4ba0:fff1:c3::1","meta.feature_category":"continuous_integration","content_length":"714","request_urgency":"low","target_duration_s":5}
{"time":"2022-05-29T08:09:06.735Z","severity":"INFO","duration_s":0.02378,"db_duration_s":0.00699,"view_duration_s":0.01679,"status":200,"method":"POST","path":"/api/v4/jobs/12730/artifacts/authorize","params":[{"key":"artifact_format","value":"zip"},{"key":"artifact_type","value":"archive"},{"key":"expire_in","value":"1 day"}],"host":"git.jan-kohnert.de","remote_ip":"127.0.0.1, 10.10.0.56","ua":"gitlab-runner 15.0.0 (15-0-stable; go1.17.7; linux/amd64)","route":"/api/:version/jobs/:id/artifacts/authorize","redis_calls":5,"redis_duration_s":0.001702,"redis_read_bytes":405,"redis_write_bytes":293,"redis_cache_calls":2,"redis_cache_duration_s":0.000796,"redis_cache_read_bytes":405,"redis_cache_write_bytes":119,"redis_shared_state_calls":3,"redis_shared_state_duration_s":0.000906,"redis_shared_state_write_bytes":174,"db_count":6,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":6,"db_main_count":6,"db_main_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_main_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_main_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.007,"db_main_duration_s":0.007,"db_main_replica_duration_s":0.0,"cpu_s":0.028257,"pid":28156,"rate_limiting_gates":[],"correlation_id":"01G47DS5W4FEAY6Q6SRX2R957F","meta.user":"jan","meta.project":"arch/klepto","meta.root_namespace":"arch","meta.client_id":"ip/10.10.0.56","meta.caller_id":"POST /api/:version/jobs/:id/artifacts/authorize","meta.remote_ip":"10.10.0.56","meta.feature_category":"build_artifacts","meta.pipeline_id":3057,"meta.job_id":12730,"content_length":"0","request_urgency":"low","target_duration_s":5}
{"time":"2022-05-29T08:09:07.591Z","severity":"INFO","duration_s":0.0269,"db_duration_s":0.00684,"view_duration_s":0.02006,"status":202,"method":"PATCH","path":"/api/v4/jobs/12730/trace","params":[],"host":"git.jan-kohnert.de","remote_ip":"127.0.0.1, 2001:4ba0:fff1:c3::1, 2001:4ba0:fff1:c3::1","ua":"gitlab-runner 15.0.0 (15-0-stable; go1.17.7; linux/amd64)","route":"/api/:version/jobs/:id/trace","queue_duration_s":0.012777,"redis_calls":7,"redis_duration_s":0.0022259999999999997,"redis_read_bytes":5,"redis_write_bytes":633,"redis_cache_calls":1,"redis_cache_duration_s":0.000325,"redis_cache_read_bytes":2,"redis_cache_write_bytes":144,"redis_shared_state_calls":6,"redis_shared_state_duration_s":0.001901,"redis_shared_state_read_bytes":3,"redis_shared_state_write_bytes":489,"db_count":7,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":7,"db_main_count":7,"db_main_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_main_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_main_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.007,"db_main_duration_s":0.007,"db_main_replica_duration_s":0.0,"cpu_s":0.032525,"pid":28215,"rate_limiting_gates":[],"correlation_id":"01G47DS6PRGS4AZ9X97AR47K2B","meta.user":"jan","meta.project":"arch/klepto","meta.root_namespace":"arch","meta.client_id":"ip/2001:4ba0:fff1:c3::1","meta.caller_id":"PATCH /api/:version/jobs/:id/trace","meta.remote_ip":"2001:4ba0:fff1:c3::1","meta.feature_category":"continuous_integration","meta.pipeline_id":3057,"meta.job_id":12730,"content_length":"341","content_range":"11821-12161","request_urgency":"low","target_duration_s":5}
{"time":"2022-05-29T08:09:09.163Z","severity":"INFO","duration_s":0.01423,"db_duration_s":0.00194,"view_duration_s":0.01229,"status":204,"method":"POST","path":"/api/v4/jobs/request","params":[{"key":"info","value":{"name":"gitlab-runner","version":"15.0.0","revision":"febb2a09","platform":"linux","architecture":"amd64","executor":"kubernetes","shell":"bash","features":{"variables":"[FILTERED]","image":null,"services":null,"artifacts":null,"cache":null,"shared":null,"upload_multiple_artifacts":null,"upload_raw_artifacts":null,"session":null,"terminal":null,"refspecs":null,"masking":null,"proxy":null,"raw_variables":"[FILTERED]","artifacts_exclude":null,"multi_build_steps":null,"trace_reset":"[FILTERED]","trace_checksum":"[FILTERED]","trace_size":"[FILTERED]","vault_secrets":"[FILTERED]","cancelable":null,"return_exit_code":null,"service_variables":"[FILTERED]"},"config":{"gpus":""}}},{"key":"token","value":"[FILTERED]"},{"key":"last_update","value":"ea81f7047ca3378fd10e27b9efbaa4f7"}],"host":"git.jan-kohnert.de","remote_ip":"127.0.0.1, 2001:4ba0:fff1:c3::1, 2001:4ba0:fff1:c3::1","ua":"gitlab-runner 15.0.0 (15-0-stable; go1.17.7; linux/amd64)","route":"/api/:version/jobs/request","queue_duration_s":0.012294,"redis_calls":9,"redis_duration_s":0.002823,"redis_read_bytes":440,"redis_write_bytes":776,"redis_cache_calls":3,"redis_cache_duration_s":0.000985,"redis_cache_read_bytes":408,"redis_cache_write_bytes":356,"redis_shared_state_calls":6,"redis_shared_state_duration_s":0.001838,"redis_shared_state_read_bytes":32,"redis_shared_state_write_bytes":420,"db_count":2,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":2,"db_main_count":2,"db_main_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_main_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_main_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.002,"db_main_duration_s":0.002,"db_main_replica_duration_s":0.0,"cpu_s":0.023876,"pid":28274,"rate_limiting_gates":[],"correlation_id":"01G47DS88FS1RD9ZP3E89B34NK","meta.client_id":"runner/7","meta.caller_id":"POST /api/:version/jobs/request","meta.remote_ip":"2001:4ba0:fff1:c3::1","meta.feature_category":"continuous_integration","content_length":"714","request_urgency":"low","target_duration_s":5}
{"time":"2022-05-29T08:09:10.322Z","severity":"INFO","duration_s":0.02284,"db_duration_s":0.0057,"view_duration_s":0.01714,"status":200,"method":"POST","path":"/api/v4/jobs/12730/artifacts/authorize","params":[{"key":"artifact_format","value":"zip"},{"key":"artifact_type","value":"archive"},{"key":"expire_in","value":"1 day"}],"host":"git.jan-kohnert.de","remote_ip":"127.0.0.1, 10.10.0.56","ua":"gitlab-runner 15.0.0 (15-0-stable; go1.17.7; linux/amd64)","route":"/api/:version/jobs/:id/artifacts/authorize","redis_calls":5,"redis_duration_s":0.001543,"redis_read_bytes":405,"redis_write_bytes":293,"redis_cache_calls":2,"redis_cache_duration_s":0.00073,"redis_cache_read_bytes":405,"redis_cache_write_bytes":119,"redis_shared_state_calls":3,"redis_shared_state_duration_s":0.000813,"redis_shared_state_write_bytes":174,"db_count":6,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":6,"db_main_count":6,"db_main_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_main_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_main_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.006,"db_main_duration_s":0.006,"db_main_replica_duration_s":0.0,"cpu_s":0.027378,"pid":28274,"rate_limiting_gates":[],"correlation_id":"01G47DS9C9773HXCS5D0S47N6Z","meta.user":"jan","meta.project":"arch/klepto","meta.root_namespace":"arch","meta.client_id":"ip/10.10.0.56","meta.caller_id":"POST /api/:version/jobs/:id/artifacts/authorize","meta.remote_ip":"10.10.0.56","meta.feature_category":"build_artifacts","meta.pipeline_id":3057,"meta.job_id":12730,"content_length":"0","request_urgency":"low","target_duration_s":5}
{"time":"2022-05-29T08:09:10.649Z","severity":"INFO","duration_s":0.03066,"db_duration_s":0.00467,"view_duration_s":0.02599,"status":202,"method":"PATCH","path":"/api/v4/jobs/12730/trace","params":[],"host":"git.jan-kohnert.de","remote_ip":"127.0.0.1, 2001:4ba0:fff1:c3::1, 2001:4ba0:fff1:c3::1","ua":"gitlab-runner 15.0.0 (15-0-stable; go1.17.7; linux/amd64)","route":"/api/:version/jobs/:id/trace","queue_duration_s":0.013152,"redis_calls":7,"redis_duration_s":0.002064,"redis_read_bytes":5,"redis_write_bytes":633,"redis_cache_calls":1,"redis_cache_duration_s":0.000386,"redis_cache_read_bytes":2,"redis_cache_write_bytes":144,"redis_shared_state_calls":6,"redis_shared_state_duration_s":0.001678,"redis_shared_state_read_bytes":3,"redis_shared_state_write_bytes":489,"db_count":7,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":7,"db_main_count":7,"db_main_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_main_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_main_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.005,"db_main_duration_s":0.005,"db_main_replica_duration_s":0.0,"cpu_s":0.039137,"pid":28215,"rate_limiting_gates":[],"correlation_id":"01G47DS9P4NZ7MQB3TPJN5BEA3","meta.user":"jan","meta.project":"arch/klepto","meta.root_namespace":"arch","meta.client_id":"ip/2001:4ba0:fff1:c3::1","meta.caller_id":"PATCH /api/:version/jobs/:id/trace","meta.remote_ip":"2001:4ba0:fff1:c3::1","meta.feature_category":"continuous_integration","meta.pipeline_id":3057,"meta.job_id":12730,"content_length":"341","content_range":"12162-12502","request_urgency":"low","target_duration_s":5}
{"time":"2022-05-29T08:09:12.155Z","severity":"INFO","duration_s":0.00954,"db_duration_s":0.00145,"view_duration_s":0.00809,"status":204,"method":"POST","path":"/api/v4/jobs/request","params":[{"key":"info","value":{"name":"gitlab-runner","version":"15.0.0","revision":"febb2a09","platform":"linux","architecture":"amd64","executor":"kubernetes","shell":"bash","features":{"variables":"[FILTERED]","image":null,"services":null,"artifacts":null,"cache":null,"shared":null,"upload_multiple_artifacts":null,"upload_raw_artifacts":null,"session":null,"terminal":null,"refspecs":null,"masking":null,"proxy":null,"raw_variables":"[FILTERED]","artifacts_exclude":null,"multi_build_steps":null,"trace_reset":"[FILTERED]","trace_checksum":"[FILTERED]","trace_size":"[FILTERED]","vault_secrets":"[FILTERED]","cancelable":null,"return_exit_code":null,"service_variables":"[FILTERED]"},"config":{"gpus":""}}},{"key":"token","value":"[FILTERED]"},{"key":"last_update","value":"ea81f7047ca3378fd10e27b9efbaa4f7"}],"host":"git.jan-kohnert.de","remote_ip":"127.0.0.1, 2001:4ba0:fff1:c3::1, 2001:4ba0:fff1:c3::1","ua":"gitlab-runner 15.0.0 (15-0-stable; go1.17.7; linux/amd64)","route":"/api/:version/jobs/request","queue_duration_s":0.008097,"redis_calls":7,"redis_duration_s":0.0013390000000000001,"redis_read_bytes":34,"redis_write_bytes":662,"redis_cache_calls":1,"redis_cache_duration_s":0.00019,"redis_cache_read_bytes":2,"redis_cache_write_bytes":242,"redis_shared_state_calls":6,"redis_shared_state_duration_s":0.001149,"redis_shared_state_read_bytes":32,"redis_shared_state_write_bytes":420,"db_count":2,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":2,"db_main_count":2,"db_main_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_main_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_main_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.002,"db_main_duration_s":0.002,"db_main_replica_duration_s":0.0,"cpu_s":0.015728,"pid":28274,"rate_limiting_gates":[],"correlation_id":"01G47DSB68K4F1BFS2W2XACP02","meta.client_id":"runner/7","meta.caller_id":"POST /api/:version/jobs/request","meta.remote_ip":"2001:4ba0:fff1:c3::1","meta.feature_category":"continuous_integration","content_length":"714","request_urgency":"low","target_duration_s":5}

Anyone having any hint on how to get at least an error message? I observed the last successful upload with gitlab version 14.10.2.

1 Like

I am also experiencing this issue after upgrading GitLab and GitLab Runners to 15.0.0.

@jankoh Did you find a solution to this issue?

@intgr good to know I’m not the only one. :smile: But unfortunately I still don’t have a solution. I’m currently still trying to debug the upload and getting a clue on what’s happening. I could not (yet) find documented changes in the API upload; I rechecked images (could have been an outdated helper image); I even tried to sniff network traffic, but still had no luck.

I created a support ticket about this, hopefully will get it resolved. Meanwhile, shall try GitLab Patch Release: 15.0.2 | GitLab

Nope, 15.0.2 did not fix the issue.

A wild guess: 15.0 removed support for background uploads:

It had been deprecated since 14.9, but I hadn’t seen that (I did see it before we went to 15.0, and we didn’t really depend on it, so I could just disable it on our 14.10).

Hmm, but I’m not using object storage backend for artifacts. They live on the filesystem.

Then my wild guess was off.

Short update: 15.0.2 does not solve the issue for me, too. Also local file storage.

The ticket I created is Artifacts upload broken after upgrade to 15.0.0 (#363701) · Issues · GitLab.org / GitLab · GitLab. Maybe we can link them or mark one as duplicate of the other…

Ah, I created a support ticket with them under our support contract which lives in a different system ( zendesk ). But in case they resolve anything there, I’ll surely paste it back here and to the ticket.

Thanks for reporting and helping each other :+1:

I’ve linked this topic in the issue and pinged our teams in the verify stage. Artifacts upload broken after upgrade to 15.0.0 (#363701) · Issues · GitLab.org / GitLab · GitLab

I’ve reproduced the problem here: Artifacts upload broken after upgrade to 15.0.0 (#363701) · Issues · GitLab.org / GitLab · GitLab

For those of you having trouble, can you show the permissions of /tmp via ls -ld /tmp? Can you make sure the sticky bit is set via chmod +t /tmp?

1 Like

That solved it for me. I checked the wrong directory in the first place, so I thought the sticky bit was set. Fixing permissions resolved the issue immediately.

1 Like

GitLab 15.1 fixes the problem, update in Artifacts upload broken after upgrade to 15.0.0 (#363701) · Issues · GitLab.org / GitLab · GitLab and backport to 15.0 coming soon.