Hi!
I have a Gitlab instance installed from the Helm charts on a k8s cluster, connected to a MinIO for object storage (using Bitnami charts and not the Gitlab included version).
We recently migrated from an Omnibus installation and everything works great except build artifacts upload (job logs work ok). Everytime a CI job tries to upload an artifact, whether it be a single file or a directory, it fails giving a 400 error with no more information.
Here is what the production.log shows:
Started POST "/api/v4/jobs/423369/artifacts?artifact_format=zip&artifact_type=archive&expire_in=1+week" for 100.64.1.44 at 2023-09-27 13:25:00 +0000
Started PATCH "/api/v4/jobs/423369/trace" for 100.64.1.224 at 2023-09-27 13:25:01 +0000
Started POST "/api/v4/jobs/request" for 100.64.0.147 at 2023-09-27 13:25:01 +0000
And the api_json.log:
{"time":"2023-09-27T13:25:00.925Z","severity":"INFO","duration_s":0.0038,"db_duration_s":0.00091,"view_duration_s":0.00289,"status":400,"method":"POST","path":"/api/v4/jobs/423369/artifacts","params":[{"key":"artifact_format","value":"zip"},{"key":"artifact_type","value":"archive"},{"key":"expire_in","value":"1 week"},{"key":"file","value":{"filename":"artifacts.zip","type":"application/octet-stream","name":"file","tempfile":null,"head":"Content-Disposition: form-data; name=\"file\"; filename=\"artifacts.zip\"\r\nContent-Type: application/octet-stream\r\n"}}],"host":"xxx-gitlab-webservice-default.gitlab-server.svc.cluster.local","remote_ip":"100.64.1.44","ua":"gitlab-runner 15.2.2 (15-2-stable; go1.17.9; linux/amd64)","route":"/api/:version/jobs/:id/artifacts","redis_calls":2,"redis_duration_s":0.000325,"redis_write_bytes":118,"redis_shared_state_calls":2,"redis_shared_state_duration_s":0.000325,"redis_shared_state_write_bytes":118,"db_count":1,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":1,"db_main_count":1,"db_main_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_main_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_main_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.001,"db_main_duration_s":0.001,"db_main_replica_duration_s":0.0,"cpu_s":0.007921,"mem_objects":4259,"mem_bytes":384184,"mem_mallocs":1139,"mem_total_bytes":554544,"pid":31,"worker_id":"puma_1","rate_limiting_gates":[],"correlation_id":"a6edb036-c83d-474c-9876-853927e4b5a6","meta.user":"xxxx","meta.project":"xxx/public-api-documentation","meta.root_namespace":"xxx","meta.client_id":"ip/100.64.1.44","meta.caller_id":"POST /api/:version/jobs/:id/artifacts","meta.remote_ip":"100.64.1.44","meta.feature_category":"build_artifacts","meta.pipeline_id":45684,"meta.job_id":423369,"content_length":"44723","request_urgency":"low","target_duration_s":5}
The object storage configuration has been verified to work on other Gitlab services.
There is no useful info in the various gitlab logs. I can see the POST request on the webservice (Iām using a Kubernetes self-hosted version of Gitlab) and the 400 status but no errors anywhere. MinIO does not show any more errors. It even seems that the request does not got to the MinIO (canāt see it when showing MinIO traces).
The only info that I found related to this issue, here or in the Gitlab issues, either are way old and do not match my issue or are not related.
Any help or pointers would be greatly appreciated!
Versions info:
- Gitlab version:
15.2.5-ee - Gitlab charts version:
6.2.5 - Bitnami MinIO charts version:
12.7.0