Timeout when uploading artifacts

GitLab CI/CD
1

Hi All,
in the last few days my ci-runner has sometimes (not always) problems to upload the artifacts with the following message:

WARNING: Uploading artifacts to coordinator... failed  id=5052 responseStatus=504 Gateway Time-out status=504 Gateway Time-out token=PZn2-nyW
WARNING: Retrying...                               
WARNING: Uploading artifacts to coordinator... failed  id=5052 responseStatus=504 Gateway Time-out status=504 Gateway Time-out token=PZn2-nyW
WARNING: Retrying...                               
WARNING: Uploading artifacts to coordinator... failed  id=5052 responseStatus=504 Gateway Time-out status=504 Gateway Time-out token=PZn2-nyW
FATAL: invalid argument                            
Job succeeded

I’m running gitlab v9.5.10, and gitlab-runner v9.4.2. both are running inside a docker container.
I visited the gitlab log files but found no evidence of this error.
How can i find out who this ‘coordinator’ is and where his log file is? Is there any place i can edit the value for this timeout?

Thanks in advance for any help.

2

can you also include your gitlab-ci.yml file in the explanation? I think something is wrong with that.
The FATAL: invalid argument hints at that.

3

Hello Raju,
I guess it can’t be the yml, because most of the time everything works out well. But once in a while there’s this Timeout.
But possibly I’m wrong, so here’s the yml:

image: registry.xxx:5000/product/build-ubuntu:16.04
before_script:

  • apt-get update -qq && apt-get install -y -qq pixz

variables:
GIT_SUBMODULE_STRATEGY: recursive

stages:

  • build
  • deploy

build:
stage: build
tags: [linux]
script:
- mkdir -p Builds/Release
- curl -L -H PRIVATE-TOKEN:xxx “https://xxx/product/3rdPartySDK/builds/artifacts/master/raw/Builds/Release/dist/3rdPartySDK.tar.bz2?job=build-16.04” -o _3rdPartySDK.tar.bz2
- cmp -s _3rdPartySDK.tar.bz2 3rdPartySDK.tar.bz2 || (mv _3rdPartySDK.tar.bz2 3rdPartySDK.tar.bz2 && tar jxf 3rdPartySDK.tar.bz2 )
- cd Builds/Release
- cmake -G Ninja …/…/
- ninja -j 32 all 2>&1 | tee -a build.log | grep -C 10 -i error
- mv dist/product.tar.xz dist/product.tar.xz
artifacts:
paths:
- Builds/Release/dist/product.tar.xz

trigger_packaging:
stage: deploy
tags: [linux]
only:
- master
- develop
script:
- “curl -X POST -F token=xxx -F ref=$CI_COMMIT_REF_NAME https://product/api/v4/projects/379/trigger/pipeline”

4

Hey Christau,

I had the exact same issue (based on the output you posted). It occurred on large files. There are several aspects I needed to adopt to solve the issue:

  1. increase the maximum file-size for job-artifacts (admin settings > Continuous Integration and Deployment > Maximum artifact size. Default is 100 MB

  2. Increase timeouts in /etc/gitlab/gitlab.rb. I´m not 100% sure which one it was, but searching for “timeout” and increasing those that default to 60s was sufficient.

  3. I´m running gitlab behind an external loadbalancer. Since artifacts are uploaded using http, I finally needed to increase the timeouts in our external loadbalancer as well.

Hope this helps. Cheers,

Mat2e

5

Thanks for your suggestions Mat2e.
But I did not succeed changing those parameters. In the meantime I upgraded to the latest gitlab-runner and never had this problem again. So it might came from a bug in the runner.
-chris-

6

Hi,

Digging up this old thread as we’ve started getting the same issue after we upgraded from 11.6.3 to 11.7.0 and then 11.8.0 yesterday (Gitlab & associated runners), what finally fixed it for us was to increase the value for unicorn[‘worker_timeout’] from 60 seconds (default) to 600 seconds (lower value would probably have solved it too).

7

Hi All

I am getting below error in pipeline
Uploading artifacts…
ivibackend/static: found 83 matching files
WARNING: Uploading artifacts to coordinator… failed id=88023 responseStatus=504 Gateway Time-out status=504 Gateway Time-out token=QUEZRKs-
WARNING: Retrying…
WARNING: Uploading artifacts to coordinator… failed id=88023 responseStatus=504 Gateway Time-out status=504 Gateway Time-out token=QUEZRKs-
WARNING: Retrying…
WARNING: Uploading artifacts to coordinator… failed id=88023 responseStatus=504 Gateway Time-out status=504 Gateway Time-out token=QUEZRKs-
FATAL: invalid argument
ERROR: Job failed: exit code 1

Also I updated the unicorn[‘worker_timeout’] = 60 to 600
in /etc/gitlab/gitlab.rb.
please let me know how I can fix this problem