Artifacts uploading results in error 500

John-H-Smith · February 28, 2024, 7:11am

Problem to solve

I installed gitlab and the runner using docker compose (using docker volumes) on linux.

docker-compose.yml

version: '3.6'
services:
  web:
    container_name: gitlab
    image: 'gitlab/gitlab-ce:16.9.1-ce.0'
    restart: always
    hostname: '[redacted]'
    environment:
      GITLAB_OMNIBUS_CONFIG: |
        gitlab_rails['smtp_enable'] = true
        gitlab_rails['smtp_address'] = "[redacted]"
        gitlab_rails['smtp_port'] = [redacted]
        gitlab_rails['smtp_user_name'] = "[redacted]"
        gitlab_rails['smtp_password'] = "[redacted]"
        gitlab_rails['smtp_domain'] = "[redacted]"
        gitlab_rails['smtp_authentication'] = "login"
        gitlab_rails['smtp_enable_starttls_auto'] = false
        gitlab_rails['smtp_tls'] = true        
        external_url 'https://[redacted]'
        gitlab_rails['gitlab_shell_ssh_port'] = 8010
        letsencrypt['enabled'] = false
        nginx['listen_port'] = 80
        nginx['listen_https'] = false
        nginx['ssl_verify_client'] = "off"
        #gitlab_rails['trusted_proxies'] = ["https://letsencrypt.org/certs/isrgrootx1.pem"]
        gitlab_rails['omniauth_enabled'] = true
        #gitlab_rails['omniauth_allow_insecure_connections'] = true
        gitlab_rails['omniauth_auto_link_user'] = ['openid_connect']
        gitlab_rails['omniauth_allow_single_sign_on'] = ['openid_connect']
        gitlab_rails['omniauth_allow_bypass_two_factor'] = ['openid_connect']
        gitlab_rails['omniauth_sync_profile_from_provider'] = ['openid_connect']
        gitlab_rails['artifacts_enabled'] = true
        gitlab_rails['object_store']['objects']['artifacts']['enabled'] = false
        # Add any other gitlab.rb configuration here, each on its own line
    ports:
      - '127.0.0.1:8008:80'
      - '127.0.0.1:8009:443'
      - '127.0.0.1:8010:22'
    volumes:
      - config:/etc/gitlab
      - logs:/var/log/gitlab
      - data:/var/opt/gitlab
#    shm_size: '256m'
    networks:
      - gitlab
  gitlab-runner:
    image: gitlab/gitlab-runner:latest
    container_name: gitlab-runner    
    restart: always
    depends_on:
      - web
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - runner:/etc/gitlab-runner
      - runner-docker-config:/root/.docker/machine
    networks:
      - gitlab
networks:
  gitlab:
    name: gitlab-network

volumes:
  runner:
  runner-docker-config:
  logs:
  config:
  data:

Runner is registered and mostly working fine.
But when trying to upload an artifact, I am getting the following error:

Job error

e[0KRunning with gitlab-runner 16.9.0 (656c1943)e[0;m
e[0K  on gitlab_runner ftdhVzvmf, system ID: r_NyEAxaWdlJXde[0;m
section_start:1709101345:prepare_executor
e[0Ke[0Ke[36;1mPreparing the "docker" executore[0;me[0;m
e[0KUsing Docker executor with image quay.io/curl/curl:latest ...e[0;m
e[0KUsing locally found image version due to "if-not-present" pull policye[0;m
e[0KUsing docker image sha256:e4684e216b8c71cac2123ee7400e301075d8318dbc62a28cc78de65ff16bf0f2 for quay.io/curl/curl:latest with digest quay.io/curl/curl@sha256:b5ef96b95b22aa91574f38fcb8b89542ffaa47bd338e9920f739806ae59b19b2 ...e[0;m
section_end:1709101346:prepare_executor
e[0Ksection_start:1709101346:prepare_script
e[0Ke[0Ke[36;1mPreparing environmente[0;me[0;m
Running on runner-ftdhvzvmf-project-4-concurrent-0 via 38a67a2bcd86...
section_end:1709101347:prepare_script
e[0Ksection_start:1709101347:get_sources
e[0Ke[0Ke[36;1mGetting source from Git repositorye[0;me[0;m
e[32;1mFetching changes with git depth set to 20...e[0;m
Reinitialized existing Git repository in /builds/root/ci-cd-testing/.git/
e[32;1mChecking out 14f73485 as detached HEAD (ref is main)...e[0;m
Removing gl-sast-sonar-report.json
Removing node_modules/

e[32;1mSkipping Git submodules setupe[0;m
section_end:1709101348:get_sources
e[0Ksection_start:1709101348:restore_cache
e[0Ke[0Ke[36;1mRestoring cachee[0;me[0;m
e[32;1mChecking cache for default-4-protected...e[0;m
No URL provided, cache will not be downloaded from shared cache server. Instead a local version of cache will be extracted.e[0;m 
e[32;1mSuccessfully extracted cachee[0;m
section_end:1709101351:restore_cache
e[0Ksection_start:1709101351:step_script
e[0Ke[0Ke[36;1mExecuting "step_script" stage of the job scripte[0;me[0;m
e[0KUsing docker image sha256:e4684e216b8c71cac2123ee7400e301075d8318dbc62a28cc78de65ff16bf0f2 for quay.io/curl/curl:latest with digest quay.io/curl/curl@sha256:b5ef96b95b22aa91574f38fcb8b89542ffaa47bd338e9920f739806ae59b19b2 ...e[0;m
e[32;1m$ curl -u "${SONAR_TOKEN}:" "${SONAR_HOST_URL}/api/issues/gitlab_sast_export?projectKey=<projectKey>&branch=${CI_COMMIT_BRANCH}&pullRequest=${CI_MERGE_REQUEST_IID}" -o gl-sast-sonar-report.jsone[0;m
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100  1774  100  1774    0     0  56679      0 --:--:-- --:--:-- --:--:-- 57225
section_end:1709101351:step_script
e[0Ksection_start:1709101351:archive_cache
e[0Ke[0Ke[36;1mSaving cache for successful jobe[0;me[0;m
e[32;1mCreating cache default-4-protected...e[0;m
node_modules/: found 20339 matching artifact files and directoriese[0;m 
No URL provided, cache will not be uploaded to shared cache server. Cache will be stored only locally.e[0;m 
e[32;1mCreated cachee[0;m
section_end:1709101356:archive_cache
e[0Ksection_start:1709101356:upload_artifacts_on_success
e[0Ke[0Ke[36;1mUploading artifacts for successful jobe[0;me[0;m
e[32;1mUploading artifacts...e[0;m
gl-sast-sonar-report.json: found 1 matching artifact files and directoriese[0;m 
e[0;33mWARNING: Uploading artifacts as "sast" to coordinator... 500 Internal Server Errore[0;m  e[0;33mide[0;m=279 e[0;33mresponseStatuse[0;m=500 Internal Server Error e[0;33mstatuse[0;m=500 e[0;33mtokene[0;m=glcbt-64
e[0;33mWARNING: Retrying...                              e[0;m  e[0;33mcontexte[0;m=artifacts-uploader e[0;33merrore[0;m=invalid argument
e[0;33mWARNING: Uploading artifacts as "sast" to coordinator... 500 Internal Server Errore[0;m  e[0;33mide[0;m=279 e[0;33mresponseStatuse[0;m=500 Internal Server Error e[0;33mstatuse[0;m=500 e[0;33mtokene[0;m=glcbt-64
e[0;33mWARNING: Retrying...                              e[0;m  e[0;33mcontexte[0;m=artifacts-uploader e[0;33merrore[0;m=invalid argument
e[0;33mWARNING: Uploading artifacts as "sast" to coordinator... 500 Internal Server Errore[0;m  e[0;33mide[0;m=279 e[0;33mresponseStatuse[0;m=500 Internal Server Error e[0;33mstatuse[0;m=500 e[0;33mtokene[0;m=glcbt-64
e[31;1mFATAL: invalid argument                           e[0;m 
section_end:1709101360:upload_artifacts_on_success
e[0Ke[31;1mERROR: Job failed: exit code 1
e[0;m

I already tried setting permissions 777 to /var/opt/gitlab/gitlab-rails/shared/artifacts/tmp/uploads/ but this isn’t working either.
Also, I added gitlab-zip-cat and gitlab-zip-metadata to /usr/local/bin as described in Error 500 when uploading artifacts through gitlab-runner for gitlab pages (#36920) · Issues · GitLab.org / GitLab FOSS · GitLab. Also not working.
I already set up the runner again.

Workhorse log says the 500 error too:

gitlab-workhorse/current

{"content_type":"application/json; charset=utf-8","correlation_id":"01HQQ890H43VYVZJB4DVTFG077","duration_ms":83,"host":"[redacted]","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"https://[redacted]/root/ci-cd-testing/-/jobs/280","remote_addr":"[redacted]:0","remote_ip":"[redacted]","route":"","status":200,"system":"http","time":"2024-02-28T07:01:18Z","ttfb_ms":83,"uri":"/root/ci-cd-testing/-/pipelines/74/stage.json?stage=vulnerability-report\u0026retried=1","user_agent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36","written_bytes":6474}
{"content_type":"text/html; charset=utf-8","correlation_id":"01HQQ891G0RZ0DKQKZX5SXDQ5G","duration_ms":3,"host":"[redacted]","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"http://[redacted]/api/v4/jobs/280/artifacts?artifact_format=raw\u0026artifact_type=sast\u0026expire_in=1+day","remote_addr":"127.0.0.1:0","remote_ip":"127.0.0.1","route":"^/api/","status":500,"system":"http","time":"2024-02-28T07:01:18Z","ttfb_ms":3,"uri":"/api/v4/jobs/280/artifacts?artifact_format=raw\u0026artifact_type=sast\u0026expire_in=1+day","user_agent":"gitlab-runner-helper 16.9.0 (16-9-stable; go1.21.7; linux/amd64)","written_bytes":3037}
{"content_type":"text/html; charset=utf-8","correlation_id":"01HQQ892AKXW4KTMBXZ236BHW7","duration_ms":94,"host":"[redacted]","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"https://[redacted]/root/ci-cd-testing/-/jobs/279","remote_addr":"[redacted]:0","remote_ip":"[redacted]","route":"","status":200,"system":"http","time":"2024-02-28T07:01:19Z","ttfb_ms":94,"uri":"/root/ci-cd-testing/-/jobs/280","user_agent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36","written_bytes":40728}
{"content_type":"text/html; charset=utf-8","correlation_id":"01HQQ892JH9XM5AQSDJ10J0TCF","duration_ms":2,"host":"[redacted]","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"http://[redacted]/api/v4/jobs/280/artifacts?artifact_format=raw\u0026artifact_type=sast\u0026expire_in=1+day","remote_addr":"127.0.0.1:0","remote_ip":"127.0.0.1","route":"^/api/","status":500,"system":"http","time":"2024-02-28T07:01:20Z","ttfb_ms":2,"uri":"/api/v4/jobs/280/artifacts?artifact_format=raw\u0026artifact_type=sast\u0026expire_in=1+day","user_agent":"gitlab-runner-helper 16.9.0 (16-9-stable; go1.21.7; linux/amd64)","written_bytes":3037}
{"content_type":"","correlation_id":"01HQQ892ZMN0K5YPDVEY5PCT87","duration_ms":11,"host":"[redacted]","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"https://[redacted]/root/ci-cd-testing/-/jobs/280","remote_addr":"[redacted]:0","remote_ip":"[redacted]","route":"^/-/","status":304,"system":"http","time":"2024-02-28T07:01:20Z","ttfb_ms":11,"uri":"/-/manifest.json","user_agent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36","written_bytes":0}
{"content_type":"application/json; charset=utf-8","correlation_id":"01HQQ892VYH8YC0HA49CQ9ENMJ","duration_ms":135,"host":"[redacted]","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"https://[redacted]/root/ci-cd-testing/-/jobs/280","remote_addr":"[redacted]:0","remote_ip":"[redacted]","route":"","status":200,"system":"http","time":"2024-02-28T07:01:20Z","ttfb_ms":134,"uri":"/root/ci-cd-testing/-/jobs/280.json","user_agent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36","written_bytes":7537}
{"content_type":"application/json; charset=utf-8","correlation_id":"01HQQ8931NTQ1KS9NXMVYPVCGM","duration_ms":64,"host":"[redacted]","level":"info","method":"POST","msg":"access","proto":"HTTP/1.1","referrer":"https://[redacted]/root/ci-cd-testing/-/jobs/280","remote_addr":"[redacted]:0","remote_ip":"[redacted]","route":"^/api/graphql\\z","status":200,"system":"http","time":"2024-02-28T07:01:20Z","ttfb_ms":64,"uri":"/api/graphql","user_agent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36","written_bytes":259}
{"content_type":"","correlation_id":"01HQQ8932H4SETR3P0P60TW8TG","duration_ms":50,"host":"[redacted]","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"https://[redacted]/root/ci-cd-testing/-/jobs/280","remote_addr":"[redacted]:0","remote_ip":"[redacted]","route":"","status":304,"system":"http","time":"2024-02-28T07:01:20Z","ttfb_ms":50,"uri":"/root/ci-cd-testing/-/jobs/280/trace","user_agent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36","written_bytes":0}
{"content_type":"","correlation_id":"01HQQ8932H4SETR3P0P72T4SFV","duration_ms":84,"host":"[redacted]","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"https://[redacted]/root/ci-cd-testing/-/jobs/280","remote_addr":"[redacted]:0","remote_ip":"[redacted]","route":"","status":304,"system":"http","time":"2024-02-28T07:01:20Z","ttfb_ms":84,"uri":"/root/ci-cd-testing/-/pipelines/74/stage.json?stage=vulnerability-report\u0026retried=1","user_agent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36","written_bytes":0}
{"content_type":"text/html; charset=utf-8","correlation_id":"01HQQ894QA0HRVD73FGER0T5EV","duration_ms":4,"host":"[redacted]","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"http://[redacted]/api/v4/jobs/280/artifacts?artifact_format=raw\u0026artifact_type=sast\u0026expire_in=1+day","remote_addr":"127.0.0.1:0","remote_ip":"127.0.0.1","route":"^/api/","status":500,"system":"http","time":"2024-02-28T07:01:22Z","ttfb_ms":4,"uri":"/api/v4/jobs/280/artifacts?artifact_format=raw\u0026artifact_type=sast\u0026expire_in=1+day","user_agent":"gitlab-runner-helper 16.9.0 (16-9-stable; go1.21.7; linux/amd64)","written_bytes":3037}

In the production.log, the following error occurs:

production.log

EOFError (EOFError):
  
config/initializers/rack_multipart_patch.rb:10:in `extract_multipart'
lib/gitlab/rack_attack/request.rb:265:in `deprecated_api_request?'
lib/gitlab/rack_attack/request.rb:195:in `throttle_unauthenticated_deprecated_api?'
lib/gitlab/rack_attack/request.rb:95:in `throttle_unauthenticated_api?'
lib/gitlab/rack_attack/request.rb:86:in `throttle?'
lib/gitlab/rack_attack.rb:139:in `block (2 levels) in configure_throttles'
lib/gitlab/middleware/read_only/controller.rb:50:in `call'
lib/gitlab/middleware/read_only.rb:18:in `call'
lib/gitlab/middleware/unauthenticated_session_expiry.rb:18:in `call'
lib/gitlab/middleware/same_site_cookies.rb:27:in `call'
lib/gitlab/middleware/path_traversal_check.rb:35:in `call'
lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'
lib/gitlab/middleware/basic_health_check.rb:25:in `call'
lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'
lib/gitlab/middleware/request_context.rb:15:in `call'
lib/gitlab/middleware/webhook_recursion_detection.rb:15:in `call'
config/initializers/fix_local_cache_middleware.rb:11:in `call'
lib/gitlab/middleware/compressed_json.rb:44:in `call'
lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call'
lib/gitlab/middleware/sidekiq_web_static.rb:20:in `call'
lib/gitlab/metrics/requests_rack_middleware.rb:79:in `call'
lib/gitlab/middleware/release_env.rb:13:in `call'
  
EOFError (EOFError):
  
config/initializers/rack_multipart_patch.rb:10:in `extract_multipart'
lib/gitlab/rack_attack/request.rb:265:in `deprecated_api_request?'
lib/gitlab/rack_attack/request.rb:195:in `throttle_unauthenticated_deprecated_api?'
lib/gitlab/rack_attack/request.rb:95:in `throttle_unauthenticated_api?'
lib/gitlab/rack_attack/request.rb:86:in `throttle?'
lib/gitlab/rack_attack.rb:139:in `block (2 levels) in configure_throttles'
lib/gitlab/middleware/read_only/controller.rb:50:in `call'
lib/gitlab/middleware/read_only.rb:18:in `call'
lib/gitlab/middleware/unauthenticated_session_expiry.rb:18:in `call'
lib/gitlab/middleware/same_site_cookies.rb:27:in `call'
lib/gitlab/middleware/path_traversal_check.rb:35:in `call'
lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'
lib/gitlab/middleware/basic_health_check.rb:25:in `call'
lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'
lib/gitlab/middleware/request_context.rb:15:in `call'
lib/gitlab/middleware/webhook_recursion_detection.rb:15:in `call'
config/initializers/fix_local_cache_middleware.rb:11:in `call'
lib/gitlab/middleware/compressed_json.rb:44:in `call'
lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call'
lib/gitlab/middleware/sidekiq_web_static.rb:20:in `call'
lib/gitlab/metrics/requests_rack_middleware.rb:79:in `call'
lib/gitlab/middleware/release_env.rb:13:in `call'
  
EOFError (EOFError):
  
config/initializers/rack_multipart_patch.rb:10:in `extract_multipart'
lib/gitlab/rack_attack/request.rb:265:in `deprecated_api_request?'
lib/gitlab/rack_attack/request.rb:195:in `throttle_unauthenticated_deprecated_api?'
lib/gitlab/rack_attack/request.rb:95:in `throttle_unauthenticated_api?'
lib/gitlab/rack_attack/request.rb:86:in `throttle?'
lib/gitlab/rack_attack.rb:139:in `block (2 levels) in configure_throttles'
lib/gitlab/middleware/read_only/controller.rb:50:in `call'
lib/gitlab/middleware/read_only.rb:18:in `call'
lib/gitlab/middleware/unauthenticated_session_expiry.rb:18:in `call'
lib/gitlab/middleware/same_site_cookies.rb:27:in `call'
lib/gitlab/middleware/path_traversal_check.rb:35:in `call'
lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'
lib/gitlab/middleware/basic_health_check.rb:25:in `call'
lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'
lib/gitlab/middleware/request_context.rb:15:in `call'
lib/gitlab/middleware/webhook_recursion_detection.rb:15:in `call'
config/initializers/fix_local_cache_middleware.rb:11:in `call'
lib/gitlab/middleware/compressed_json.rb:44:in `call'
lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call'
lib/gitlab/middleware/sidekiq_web_static.rb:20:in `call'
lib/gitlab/metrics/requests_rack_middleware.rb:79:in `call'
lib/gitlab/middleware/release_env.rb:13:in `call'

Right now, I have no more ideas on how to get that working.
Do you?
Thanks in advance!

Versions

Self-managed
GitLab.com SaaS

Versions

GitLab: gitlab-ce:16.9.1

John-H-Smith · March 2, 2024, 7:52am

No ideas so far?

bblegoman · March 2, 2024, 9:29pm

Just faced the same. Did you solve it, @John-H-Smith ?

John-H-Smith · March 4, 2024, 6:18am

Nope, still no solution from my side.

dnsmichi · March 4, 2024, 8:00pm

This sounds like a bug to me, in the artifact-uploader ↔ server connection, especially with HTTP 500 and invalid argument errors. Please create a bug report issue

John-H-Smith · March 5, 2024, 6:50am

I just noted there are several issues for this specific problem. But all of them are open:

So seems like it is a very common problem.

bblegoman · March 5, 2024, 6:52am

Already did it, but label “bug” is missing, hope it won’t be the problem

dnsmichi · March 5, 2024, 6:13pm

Thanks, I added labels and a comment linking back here.