Error with uploading Artifacts 500

GitLab CI/CD
1

Problem to solve

I am trying to run a gitlab CI/CD on 17.3.1 and seem to be catching an issue where artifacts that I define are not being uploaded.

Runner log shows this:

test.txt: found 1 matching artifact files and directories 
WARNING: Uploading artifacts as "archive" to coordinator... 500 Internal Server Error  id=622 responseStatus=500 Internal Server Error status=500 token=glcbt-64
WARNING: Retrying...                                context=artifacts-uploader error=invalid argument
WARNING: Uploading artifacts as "archive" to coordinator... 500 Internal Server Error  id=622 responseStatus=500 Internal Server Error status=500 token=glcbt-64
WARNING: Retrying...                                context=artifacts-uploader error=invalid argument
WARNING: Uploading artifacts as "archive" to coordinator... 500 Internal Server Error  id=622 responseStatus=500 Internal Server Error status=500 token=glcbt-64

I am seeing this in the error log:

024-08-21T20:21:36.785495862Z ==> /var/log/gitlab/nginx/gitlab_access.log <==
2024-08-21T20:21:36.785500637Z 10.0.1.40 - - [21/Aug/2024:20:21:36 +0000] "GET /api/v4/jobs/620/artifacts?artifact_format=zip&artifact_type=archive&expire_in=1+week HTTP/1.0" 500 1624 "http://gitlab.bitformations.ca/api/v4/jobs/620/artifacts" "gitlab-runner-helper 17.3.1 (17-3-stable; go1.22.5; linux/amd64)" -
2024-08-21T20:21:38.950487311Z 
2024-08-21T20:21:38.950519951Z ==> /var/log/gitlab/gitlab-rails/production.log <==
2024-08-21T20:21:38.950524451Z   
2024-08-21T20:21:38.950527208Z EOFError (EOFError):
2024-08-21T20:21:38.950530544Z   
2024-08-21T20:21:38.950534569Z rack (2.2.9) lib/rack/multipart/parser.rb:379:in `handle_empty_content!'
2024-08-21T20:21:38.950538797Z rack (2.2.9) lib/rack/multipart/parser.rb:200:in `on_read'
2024-08-21T20:21:38.950542449Z rack (2.2.9) lib/rack/multipart/parser.rb:76:in `parse'
2024-08-21T20:21:38.950546012Z rack (2.2.9) lib/rack/multipart.rb:53:in `extract_multipart'
2024-08-21T20:21:38.950550148Z config/initializers/rack_multipart_patch.rb:10:in `extract_multipart'
2024-08-21T20:21:38.950554219Z rack (2.2.9) lib/rack/request.rb:594:in `parse_multipart'
2024-08-21T20:21:38.950558285Z rack (2.2.9) lib/rack/request.rb:446:in `POST'
2024-08-21T20:21:38.950562270Z rack (2.2.9) lib/rack/request.rb:469:in `params'
2024-08-21T20:21:38.950565825Z rack (2.2.9) lib/rack/request.rb:32:in `params'
2024-08-21T20:21:38.950569160Z lib/gitlab/rack_attack/request.rb:272:in `deprecated_api_request?'
2024-08-21T20:21:38.950572665Z lib/gitlab/rack_attack/request.rb:198:in `throttle_unauthenticated_deprecated_api?'
2024-08-21T20:21:38.950577004Z lib/gitlab/rack_attack/request.rb:91:in `throttle_unauthenticated_api?'
2024-08-21T20:21:38.950580000Z lib/gitlab/rack_attack/request.rb:82:in `throttle?'
2024-08-21T20:21:38.950582937Z lib/gitlab/rack_attack.rb:149:in `block (2 levels) in configure_throttles'
2024-08-21T20:21:38.950585897Z rack-attack (6.7.0) lib/rack/attack/throttle.rb:53:in `discriminator_for'
2024-08-21T20:21:38.950599533Z rack-attack (6.7.0) lib/rack/attack/throttle.rb:26:in `matched_by?'
2024-08-21T20:21:38.950602393Z rack-attack (6.7.0) lib/rack/attack/configuration.rb:91:in `block in throttled?'
2024-08-21T20:21:38.950605192Z rack-attack (6.7.0) lib/rack/attack/configuration.rb:90:in `any?'
2024-08-21T20:21:38.950607925Z rack-attack (6.7.0) lib/rack/attack/configuration.rb:90:in `throttled?'
2024-08-21T20:21:38.950610651Z rack-attack (6.7.0) lib/rack/attack.rb:118:in `call'
2024-08-21T20:21:38.950613506Z warden (1.2.9) lib/warden/manager.rb:36:in `block in call'
2024-08-21T20:21:38.950616375Z warden (1.2.9) lib/warden/manager.rb:34:in `catch'
2024-08-21T20:21:38.950619177Z warden (1.2.9) lib/warden/manager.rb:34:in `call'
2024-08-21T20:21:38.950621826Z rack-cors (2.0.2) lib/rack/cors.rb:102:in `call'
2024-08-21T20:21:38.950624607Z rack (2.2.9) lib/rack/tempfile_reaper.rb:15:in `call'
2024-08-21T20:21:38.950627467Z rack (2.2.9) lib/rack/etag.rb:27:in `call'
2024-08-21T20:21:38.950630197Z rack (2.2.9) lib/rack/conditional_get.rb:27:in `call'
2024-08-21T20:21:38.950632943Z rack (2.2.9) lib/rack/head.rb:12:in `call'
2024-08-21T20:21:38.950635669Z actionpack (7.0.8.4) lib/action_dispatch/http/permissions_policy.rb:38:in `call'
2024-08-21T20:21:38.950638543Z actionpack (7.0.8.4) lib/action_dispatch/http/content_security_policy.rb:36:in `call'
2024-08-21T20:21:38.950641293Z lib/gitlab/middleware/read_only/controller.rb:50:in `call'
2024-08-21T20:21:38.950643995Z lib/gitlab/middleware/read_only.rb:18:in `call'
2024-08-21T20:21:38.950647947Z lib/gitlab/middleware/unauthenticated_session_expiry.rb:18:in `call'
2024-08-21T20:21:38.950650776Z rack (2.2.9) lib/rack/session/abstract/id.rb:266:in `context'
2024-08-21T20:21:38.950653524Z rack (2.2.9) lib/rack/session/abstract/id.rb:260:in `call'
2024-08-21T20:21:38.950656501Z actionpack (7.0.8.4) lib/action_dispatch/middleware/cookies.rb:704:in `call'
2024-08-21T20:21:38.950659221Z lib/gitlab/middleware/same_site_cookies.rb:27:in `call'
2024-08-21T20:21:38.950661905Z actionpack (7.0.8.4) lib/action_dispatch/middleware/callbacks.rb:27:in `block in call'
2024-08-21T20:21:38.950664665Z activesupport (7.0.8.4) lib/active_support/callbacks.rb:99:in `run_callbacks'
2024-08-21T20:21:38.950667401Z actionpack (7.0.8.4) lib/action_dispatch/middleware/callbacks.rb:26:in `call'
2024-08-21T20:21:38.950670119Z sentry-rails (5.18.2) lib/sentry/rails/rescued_exception_interceptor.rb:12:in `call'
2024-08-21T20:21:38.950672897Z actionpack (7.0.8.4) lib/action_dispatch/middleware/debug_exceptions.rb:28:in `call'
2024-08-21T20:21:38.950675628Z lib/gitlab/middleware/path_traversal_check.rb:27:in `call'
2024-08-21T20:21:38.950678316Z lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'
2024-08-21T20:21:38.950681012Z sentry-ruby (5.18.2) lib/sentry/rack/capture_exceptions.rb:30:in `block (2 levels) in call'
2024-08-21T20:21:38.950687241Z sentry-ruby (5.18.2) lib/sentry/hub.rb:258:in `with_session_tracking'
2024-08-21T20:21:38.950690010Z sentry-ruby (5.18.2) lib/sentry-ruby.rb:404:in `with_session_tracking'
2024-08-21T20:21:38.950692733Z sentry-ruby (5.18.2) lib/sentry/rack/capture_exceptions.rb:21:in `block in call'
2024-08-21T20:21:38.950695442Z sentry-ruby (5.18.2) lib/sentry/hub.rb:59:in `with_scope'
2024-08-21T20:21:38.950698170Z sentry-ruby (5.18.2) lib/sentry-ruby.rb:384:in `with_scope'
2024-08-21T20:21:38.950700877Z sentry-ruby (5.18.2) lib/sentry/rack/capture_exceptions.rb:20:in `call'
2024-08-21T20:21:38.950703605Z actionpack (7.0.8.4) lib/action_dispatch/middleware/show_exceptions.rb:29:in `call'
2024-08-21T20:21:38.950706367Z lib/gitlab/middleware/basic_health_check.rb:25:in `call'
2024-08-21T20:21:38.950709209Z lograge (0.11.2) lib/lograge/rails_ext/rack/logger.rb:15:in `call_app'
2024-08-21T20:21:38.950711990Z railties (7.0.8.4) lib/rails/rack/logger.rb:25:in `block in call'
2024-08-21T20:21:38.950714677Z activesupport (7.0.8.4) lib/active_support/tagged_logging.rb:99:in `block in tagged'
2024-08-21T20:21:38.950717383Z activesupport (7.0.8.4) lib/active_support/tagged_logging.rb:37:in `tagged'
2024-08-21T20:21:38.950720147Z activesupport (7.0.8.4) lib/active_support/tagged_logging.rb:99:in `tagged'
2024-08-21T20:21:38.950723078Z railties (7.0.8.4) lib/rails/rack/logger.rb:25:in `call'
2024-08-21T20:21:38.950725758Z actionpack (7.0.8.4) lib/action_dispatch/middleware/remote_ip.rb:93:in `call'
2024-08-21T20:21:38.950728716Z lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'
2024-08-21T20:21:38.950731438Z lib/gitlab/middleware/request_context.rb:15:in `call'
2024-08-21T20:21:38.950734128Z lib/gitlab/middleware/webhook_recursion_detection.rb:15:in `call'
2024-08-21T20:21:38.950736834Z request_store (1.5.1) lib/request_store/middleware.rb:19:in `call'
2024-08-21T20:21:38.950739532Z rack (2.2.9) lib/rack/method_override.rb:24:in `call'
2024-08-21T20:21:38.950742261Z rack (2.2.9) lib/rack/runtime.rb:22:in `call'
2024-08-21T20:21:38.950744926Z rack-timeout (0.7.0) lib/rack/timeout/core.rb:154:in `block in call'
2024-08-21T20:21:38.950747672Z rack-timeout (0.7.0) lib/rack/timeout/support/timeout.rb:19:in `timeout'
2024-08-21T20:21:38.950750676Z rack-timeout (0.7.0) lib/rack/timeout/core.rb:153:in `call'
2024-08-21T20:21:38.950753416Z config/initializers/fix_local_cache_middleware.rb:11:in `call'
2024-08-21T20:21:38.950756207Z lib/gitlab/middleware/compressed_json.rb:44:in `call'
2024-08-21T20:21:38.950758927Z actionpack (7.0.8.4) lib/action_dispatch/middleware/executor.rb:14:in `call'
2024-08-21T20:21:38.950761646Z lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call'
2024-08-21T20:21:38.950764418Z rack (2.2.9) lib/rack/sendfile.rb:110:in `call'
2024-08-21T20:21:38.950767139Z lib/gitlab/middleware/sidekiq_web_static.rb:20:in `call'
2024-08-21T20:21:38.950773744Z lib/gitlab/metrics/requests_rack_middleware.rb:79:in `call'
2024-08-21T20:21:38.950776522Z gitlab-labkit (0.36.1) lib/labkit/middleware/rack.rb:22:in `block in call'
2024-08-21T20:21:38.950779234Z gitlab-labkit (0.36.1) lib/labkit/context.rb:35:in `with_context'
2024-08-21T20:21:38.950781954Z gitlab-labkit (0.36.1) lib/labkit/middleware/rack.rb:21:in `call'
2024-08-21T20:21:38.950784650Z actionpack (7.0.8.4) lib/action_dispatch/middleware/request_id.rb:26:in `call'
2024-08-21T20:21:38.950787378Z actionpack (7.0.8.4) lib/action_dispatch/middleware/host_authorization.rb:131:in `call'
2024-08-21T20:21:38.950790201Z railties (7.0.8.4) lib/rails/engine.rb:530:in `call'
2024-08-21T20:21:38.950793014Z railties (7.0.8.4) lib/rails/railtie.rb:226:in `public_send'
2024-08-21T20:21:38.950795888Z railties (7.0.8.4) lib/rails/railtie.rb:226:in `method_missing'
2024-08-21T20:21:38.950798612Z lib/gitlab/middleware/release_env.rb:13:in `call'
2024-08-21T20:21:38.950801571Z rack (2.2.9) lib/rack/urlmap.rb:74:in `block in call'
2024-08-21T20:21:38.950804281Z rack (2.2.9) lib/rack/urlmap.rb:58:in `each'
2024-08-21T20:21:38.950811272Z rack (2.2.9) lib/rack/urlmap.rb:58:in `call'
2024-08-21T20:21:38.950814240Z puma (6.4.0) lib/puma/configuration.rb:272:in `call'
2024-08-21T20:21:38.950817766Z puma (6.4.0) lib/puma/request.rb:100:in `block in handle_request'
2024-08-21T20:21:38.950820618Z puma (6.4.0) lib/puma/thread_pool.rb:378:in `with_force_shutdown'
2024-08-21T20:21:38.950823402Z puma (6.4.0) lib/puma/request.rb:99:in `handle_request'
2024-08-21T20:21:38.950826224Z puma (6.4.0) lib/puma/server.rb:443:in `process_client'
2024-08-21T20:21:38.950829114Z puma (6.4.0) lib/puma/server.rb:241:in `block in run'
2024-08-21T20:21:38.950832202Z puma (6.4.0) lib/puma/thread_pool.rb:155:in `block in spawn_thread'

Steps to reproduce

Use a Basic configuration:

build:
  image: node:lts 
  stage: build
  script:
    - echo "Building..." > test.txt
    - npm install
    - npm run build
    # - tar -czf /tmp/site.tar.gz -C public .
  artifacts:
    expire_in: 1 week
    paths:
      - test.txt
  only:
    - main

Versions

Please select whether options apply, and add the version information.

  • Self-managed
  • GitLab.com SaaS
  • Self-hosted Runners

Versions

Helpful resources

  1. Before opening a new topic, make sure to search for keywords in the forum search
  2. Check the GitLab Runner and GitLab projects for existing issues. If you encounter a bug, please create a bug report issue.
  3. Troubleshooting docs: GitLab Runner, self-managed GitLab instances.

Thanks for taking the time to be thorough in your request, it really helps! :blush:

2

Sounds like Uploading artifacts fails with 500 error (#444271) · Issues · GitLab.org / GitLab · GitLab Can you add your findings over there, too?

3

We updated two installations from 17.2.2 to 17.3.1, this broke gitaly on both installations and lead to error 500 all over the place, including the UI. We rolled back both machines.