500 Internal error when trying to upload artifacts

GitLab CI/CD
, ,
1

Hello,
I’m running a self hosted gitlab-ce server (v13.4.2) in a docker container on ubuntu and have a ci runner (v13.4.1) on a windows server.

I have a test job running
with a pretty simple config:

stages:
  - build-test
build_test_job:
  stage: build-test
  only:
- branches
  script:
- '& "$env:NUGET_PATH" restore'
- '& "$env:MSBUILD_PATH" /p:Configuration=Release /clp:ErrorsOnly'
- '& "$env:NUNIT_PATH\nunit3-console.exe" "$env:UNITTEST_FOLDER\UnitTest.Tests.dll"'
  dependencies:
  artifacts:
expire_in: 2 days
paths:
  - '$env:UNITTEST_FOLDER'

Everything passes ok until it gets to the stage where it goes to upload the artifacts I get this error message:

image
image1008×235 10.8 KB

Looking at the production log I found this which may or may not be related:

image
image801×261 13.2 KB

I also did a grep -r 'status":500 . on /var/log/gitlab/gitlab-workhorse and came up with this:

./@400000005f7d685b0063ccd4.u:{"correlation_id":"iUbOPOT3CU7","error":"handleFileUploads: extract files from multipart: persisting multipart file: uploadLocalFile: create file: open /var/opt/gitlab/gitlab-rails/shared/artifacts/tmp/uploads/artifacts.zip583931933: permission denied","level":"error","method":"POST","msg":"error","time":"2020-10-06T19:31:03Z","uri":"/api/v4/jobs/454/artifacts?artifact_format=zip\u0026artifact_type=archive\u0026expire_in=2+days"}

which leads me to believe there is some permission issue when trying to upload artifacts from the runner to the gitlab server, but I don’t even know where to begin to start figuring out how to solve this. Any help would be greatly appreciated.

2

what are the permissions on that directory ( ls -al /var/opt/gitlab/gitlab-rails/shared/artifacts/tmp/uploads/)?
If it is not owned/writable for the user that runs Gitlab, you can change the ownership with chown or set the correct permissions with chmod (but don’t use chmod 777).
Another culprit might be selinux or apparmor if you got that installed, but I’m not a wizard on those terrains.

3

Tried changing the permissions to test to 777 for that directory and still get the same error. I think because permission is denied on each artifact that gets created, but since that is a new file each time the runner runs, I don’t really know how I could change permissions so those would be allowed. I also checked apt list --installed and in the docker image se-linux or apparmor are not installed.

4

Oh just for the check: is the whole path accessible for the user that runs gitlab?

5

yeah tried that too and didn’t seem to change anything

6

Hello.

I had the same problem with uploading artifacts to GitLab that runs on Windows Docker.

Instead of using bind-mounts for the data/config/log volumes in the gitlab container, which apparently cause the problem, switch to docker volumes.

Doesn’t works:
volumes:
- ‘$PATH_TO_VOLUME_CONFIG:/etc/gitlab’
- ‘$PATH_TO_VOLUME_LOG:/var/log/gitlab’
- ‘$PATH_TO_VOLUME_DATA:/var/opt/gitlab’

Works:
volumes:
gitlab-runner:
driver: local
gitlab-config:
driver: local
gitlab-logs:
driver: local
gitlab-data:
driver: local

volumes:
- ‘gitlab-config:/etc/gitlab’
- ‘gitlab-logs:/var/log/gitlab’
- ‘gitlab-data:/var/opt/gitlab’

1 Like
7

Thank you you saved me hours with this