Problem uploading artifacts to S3 (direct_upload) - cache works, artifacts not

Hi to everyone,
we are using gitlab in our own server (version 13.6.3, Ubuntu 16.04) and jobs running over gitlab-runner 13.5.0 on the same machine
We want to upload directly all our artifacts and cache to our own hosted s3 storage, cache is uploading ok but artifacts are returning 502 bad gateway.
With direct_upload=false artifacts are uploading corretly to out gitlab server but with direct_upload=true we get that 502 bad gateway.

Say also that we are working under a proxy.

We set in /etc/gitlab/gitlab.rb as:

 gitlab_rails['artifacts_object_store_enabled'] = true
 gitlab_rails['artifacts_object_store_direct_upload'] = true
# gitlab_rails['artifacts_object_store_background_upload'] = true
 gitlab_rails['artifacts_object_store_proxy_download'] = true
 gitlab_rails['artifacts_object_store_remote_directory'] = "artifacts"

I have made a easy example to show what is happening

gitlab-ci.yaml:

stages:
  - build_stage
 
build:
  stage: build_stage
  script:
    - echo "building..." >> ./build_result.txt
  cache:
    key: ${CI_COMMIT_REF_SLUG}
    paths:
      - ./build_result.txt
  artifacts:
    paths:
    - build_result.txt
    expire_in: 1 week
  tags:
    - cluster 

and the result is :

e[0KRunning with gitlab-runner 13.5.0 (ece86343)
e[0;me[0K  on Ubuntu-16 4nx1PMCW
e[0;msection_start:1609835429:prepare_executor
e[0Ke[0Ke[36;1mPreparing the "shell" executore[0;m
e[0;me[0KUsing Shell executor...
e[0;msection_end:1609835429:prepare_executor
e[0Ksection_start:1609835429:prepare_script
e[0Ke[0Ke[36;1mPreparing environmente[0;m
e[0;mRunning on Ubuntu-16...
section_end:1609835430:prepare_script
e[0Ksection_start:1609835430:get_sources
e[0Ke[0Ke[36;1mGetting source from Git repositorye[0;m
e[0;me[32;1mFetching changes with git depth set to 50...e[0;m
Reinitialized existing Git repository in /home/gitlab-runner/builds/4nx1PMCW/0/root/test/.git/
e[32;1mChecking out e8a8750c as master...e[0;m
Removing build_result.txt
e[32;1mSkipping Git submodules setupe[0;m
section_end:1609835430:get_sources
e[0Ksection_start:1609835430:restore_cache
e[0Ke[0Ke[36;1mRestoring cachee[0;m
e[0;me[32;1mChecking cache for master-11...e[0;m
Runtime platform                                  e[0;m  arche[0;m=amd64 ose[0;m=linux pide[0;m=17740 revisione[0;m=ece86343 versione[0;m=13.5.0
Downloading cache.zip from https://ours3server.domain/runner/cache/runner/4nx1PMCW/project/4/master-11e[0;m 
e[32;1mSuccessfully extracted cachee[0;m
section_end:1609835431:restore_cache
e[0Ksection_start:1609835431:step_script
e[0Ke[0Ke[36;1mExecuting "step_script" stage of the job scripte[0;m
e[0;me[32;1m$ echo "building..." >> ./build_result.txte[0;m
section_end:1609835431:step_script
e[0Ksection_start:1609835431:archive_cache
e[0Ke[0Ke[36;1mSaving cache for successful jobe[0;m
e[0;me[32;1mCreating cache master-11...e[0;m
Runtime platform                                  e[0;m  arche[0;m=amd64 ose[0;m=linux pide[0;m=17790 revisione[0;m=ece86343 versione[0;m=13.5.0
./build_result.txt: found 1 matching files and directoriese[0;m 
Uploading cache.zip to https://ours3server.domain/runner/cache/runner/4nx1PMCW/project/4/master-11e[0;m 
e[32;1mCreated cachee[0;m
section_end:1609835434:archive_cache
e[0Ksection_start:1609835434:upload_artifacts_on_success
e[0Ke[0Ke[36;1mUploading artifacts for successful jobe[0;m
e[0;me[32;1mUploading artifacts...e[0;m
Runtime platform                                  e[0;m  arche[0;m=amd64 ose[0;m=linux pide[0;m=17865 revisione[0;m=ece86343 versione[0;m=13.5.0
build_result.txt: found 1 matching files and directoriese[0;m 
**e[0;33mWARNING: Uploading artifacts as "archive" to coordinator... failede[0;m  e[0;33mide[0;m=67 e[0;33mresponseStatuse[0;m=500 Internal Server Error e[0;33mstatuse[0;m=500 e[0;33mtokene[0;m=_8q6av1B**
**e[0;33mWARNING: Retrying...                              e[0;m  e[0;33mcontexte[0;m=artifacts-uploader e[0;33merrore[0;m=invalid argument**
**e[0;33mWARNING: Uploading artifacts as "archive" to coordinator... failede[0;m  e[0;33mide[0;m=67 e[0;33mresponseStatuse[0;m=502 Bad Gateway e[0;33mstatuse[0;m=502 e[0;33mtokene[0;m=_8q6av1B**
**e[0;33mWARNING: Retrying...                              e[0;m  e[0;33mcontexte[0;m=artifacts-uploader e[0;33merrore[0;m=invalid argument**
**e[0;33mWARNING: Uploading artifacts as "archive" to coordinator... failede[0;m  e[0;33mide[0;m=67 e[0;33mresponseStatuse[0;m=502 Bad Gateway e[0;33mstatuse[0;m=502** e[0;33mtokene[0;m=_8q6av1B
e[31;1mFATAL: invalid argument                           e[0;m 
section_end:1609835607:upload_artifacts_on_success
e[0Ksection_start:1609835607:cleanup_file_variables
e[0Ke[0Ke[36;1mCleaning up file based variablese[0;m
e[0;msection_end:1609835607:cleanup_file_variables
e[0Ke[31;1mERROR: Job failed: exit status 1
e[0;m

Could anyone help me to know whats happening?

Thank you very much
Joan Caparrós

@JoanCSUC it seems there was some confusion intending to remove this topic, as it landed in the moderation queue. I’ll go ahead and mark it as resolved as per your comment in the queue. Thanks for participating in the forum!

Issue resolved, I had a problem with my proxy

where is the documentation to setup a local S3 (minimio) storage as the artifacts repository? I copied the four settings this post to gitlab.rb but got an error running reconfigure. So I’m guessing there’s more involved but I’m not finding anything on the ‘artifacts’ documentation page.

I have three servers, each running gitlab-runner, and want them to share artifacts so I don’t have to designate specific projects to specific runners and still ensure pipelines work properly across the gitlab-runner cluster

Thanks