Downloading artifacts for compiling (3498)…FATAL: permission denied
Sometimes, but not always, it might happen that a job fails because it can’t download the artifacts.
When it fails, following lines are logged in job:
Running with gitlab-runner 15.2.0 (7f093137)
on docker-runner runner35.my.server D4ikeX8k
Preparing the "docker" executor
00:03
Using Docker executor with image registry.gitlab.com/gitlab-org/release-cli:latest ...
Using locally found image version due to "if-not-present" pull policy
Using docker image sha256:ebe96f5ba4f0af772489cc38a4a3861a6c8156d46e7a35f9dfbb6c520c44463a for registry.gitlab.com/gitlab-org/release-cli:latest with digest registry.gitlab.com/gitlab-org/release-cli@sha256:af74927a13c44668cf3741a8e3ef513cd4353703707c64b51b6f6ea98d3fefaf ...
Preparing environment
00:01
Running on runner-d4ikex8k-project-93-concurrent-0 via 5e90c7370db8...
Getting source from Git repository
00:04
Fetching changes with git depth set to 50...
Reinitialized existing Git repository in /builds/CompuMaster/bierdeckel-automation/.git/
[...]
Downloading artifacts
00:02
Downloading artifacts for compiling (3498)...
ERROR: Downloading artifacts from coordinator... forbidden id=3505 responseStatus=403 Forbidden status=GET http://gitlab.my.server/api/v4/jobs/3505/artifacts: 403 Forbidden token=--id
FATAL: permission denied
ERROR: Job failed: exit code 1
If I restart this job again, it usually runs perfectly without any problems.
The artifacts download error occurs typically on random jobs. Sometimes, a pipeline with 13 jobs succeeds, sometimes, the same pipeline fails because one of the jobs failed downloading the artifacts.
-
What version are you on? Are you using self-managed or GitLab.com? Running GitLab server+runner on my own virtual machines with Ubuntu 20.04
-
GitLab (Hint:
/help): 15.2.2-ee.0
-
Runner (Hint:
/admin/runners): 15.2.0 (7f093137)
Hi @jochenwezel
I would suggest to start digging in GitLab logs to find any hints what’s causing it.
Any hint where to start in logs? I haven’t found anything, yet 
Upgraded to latest version right now, after that, I retried another pipeline. gitlab logs found with following lines.
job artifacts POST for job 3524 was successful (HTTP status code 204), while job artifacts GET failed for job 3521 with HTTP status code 403
./nginx/gitlab_access.log:192.168.7.21 - - [29/Aug/2022:13:18:29 +0200] "GET /CompuMaster/bierdeckel-automation.git/info/refs?service=git-upload-pack HTTP/1.1" 401 26 "" "git/2.35.4.windows.1" -
./nginx/gitlab_access.log:192.168.7.208 - - [29/Aug/2022:13:18:29 +0200] "POST /api/v4/jobs/3524/artifacts?artifact_format=zip&artifact_type=archive&expire_in=1+month HTTP/1.1" 201 3 "" "gitlab-runner 15.2.0 (15-2-stable; go1.17.9; windows/amd64)" -
./nginx/gitlab_access.log:192.168.6.224 - - [29/Aug/2022:13:18:29 +0200] "POST /api/v4/jobs/request HTTP/1.1" 204 0 "" "gitlab-runner 15.2.0 (15-2-stable; go1.17.9; windows/amd64)" -
./nginx/gitlab_access.log:192.168.6.58 - - [29/Aug/2022:13:18:29 +0200] "POST /api/v4/jobs/request HTTP/1.1" 204 0 "" "gitlab-runner 15.2.0 (15-2-stable; go1.17.9; linux/amd64)" -
./nginx/gitlab_access.log:192.168.7.208 - - [29/Aug/2022:13:18:29 +0200] "POST /api/v4/jobs/3524/artifacts?artifact_format=gzip&artifact_type=junit&expire_in=1+month HTTP/1.1" 201 3 "" "gitlab-runner 15.2.0 (15-2-stable; go1.17.9; windows/amd64)" -
./nginx/gitlab_access.log:192.168.7.208 - - [29/Aug/2022:13:18:30 +0200] "PATCH /api/v4/jobs/3524/trace HTTP/1.1" 202 2 "" "gitlab-runner 15.2.0 (15-2-stable; go1.17.9; windows/amd64)" -
./nginx/gitlab_access.log:192.168.7.21 - jwezel@compumaster.de [29/Aug/2022:13:18:30 +0200] "GET /CompuMaster/bierdeckel-automation.git/info/refs?service=git-upload-pack HTTP/1.1" 200 164 "" "git/2.35.4.windows.1" -
./nginx/gitlab_access.log:192.168.7.208 - - [29/Aug/2022:13:18:30 +0200] "PUT /api/v4/jobs/3524 HTTP/1.1" 200 3 "" "gitlab-runner 15.2.0 (15-2-stable; go1.17.9; windows/amd64)" -
./nginx/gitlab_access.log:192.168.7.208 - - [29/Aug/2022:13:18:30 +0200] "POST /api/v4/jobs/request HTTP/1.1" 204 0 "" "gitlab-runner 15.2.0 (15-2-stable; go1.17.9; windows/amd64)" -
./nginx/gitlab_access.log:192.168.6.62 - - [29/Aug/2022:13:18:30 +0200] "POST /api/v4/jobs/request HTTP/1.1" 204 0 "" "gitlab-runner 15.2.0 (15-2-stable; go1.17.9; linux/amd64)" -
./nginx/gitlab_access.log:192.168.6.224 - - [29/Aug/2022:13:18:31 +0200] "POST /api/v4/jobs/request HTTP/1.1" 204 0 "" "gitlab-runner 15.2.0 (15-2-stable; go1.17.9; windows/amd64)" -
./nginx/gitlab_access.log:192.168.7.21 - jwezel@compumaster.de [29/Aug/2022:13:18:31 +0200] "POST /CompuMaster/bierdeckel-automation.git/git-upload-pack HTTP/1.1" 200 5773 "" "git/2.35.4.windows.1" -
./nginx/gitlab_access.log:192.168.6.58 - - [29/Aug/2022:13:18:31 +0200] "GET /api/v4/jobs/3521/artifacts?direct_download=true HTTP/1.1" 403 27 "" "Go-http-client/1.1" -
./nginx/gitlab_access.log:192.168.6.224 - - [29/Aug/2022:13:18:32 +0200] "POST /api/v4/jobs/request HTTP/1.1" 204 0 "" "gitlab-runner 15.2.0 (15-2-stable; go1.17.9; windows/amd64)" -