500 Internal error when trying to upload artifacts

I’m running a self hosted gitlab-ce server (v13.4.2) in a docker container on ubuntu and have a ci runner (v13.4.1) on a windows server.

I have a test job running
with a pretty simple config:

  - build-test
  stage: build-test
- branches
- '& "$env:NUGET_PATH" restore'
- '& "$env:MSBUILD_PATH" /p:Configuration=Release /clp:ErrorsOnly'
- '& "$env:NUNIT_PATH\nunit3-console.exe" "$env:UNITTEST_FOLDER\UnitTest.Tests.dll"'
expire_in: 2 days

Everything passes ok until it gets to the stage where it goes to upload the artifacts I get this error message:

Looking at the production log I found this which may or may not be related:

I also did a grep -r 'status":500 . on /var/log/gitlab/gitlab-workhorse and came up with this:

./@400000005f7d685b0063ccd4.u:{"correlation_id":"iUbOPOT3CU7","error":"handleFileUploads: extract files from multipart: persisting multipart file: uploadLocalFile: create file: open /var/opt/gitlab/gitlab-rails/shared/artifacts/tmp/uploads/artifacts.zip583931933: permission denied","level":"error","method":"POST","msg":"error","time":"2020-10-06T19:31:03Z","uri":"/api/v4/jobs/454/artifacts?artifact_format=zip\u0026artifact_type=archive\u0026expire_in=2+days"}

which leads me to believe there is some permission issue when trying to upload artifacts from the runner to the gitlab server, but I don’t even know where to begin to start figuring out how to solve this. Any help would be greatly appreciated.

what are the permissions on that directory ( ls -al /var/opt/gitlab/gitlab-rails/shared/artifacts/tmp/uploads/)?
If it is not owned/writable for the user that runs Gitlab, you can change the ownership with chown or set the correct permissions with chmod (but don’t use chmod 777).
Another culprit might be selinux or apparmor if you got that installed, but I’m not a wizard on those terrains.

Tried changing the permissions to test to 777 for that directory and still get the same error. I think because permission is denied on each artifact that gets created, but since that is a new file each time the runner runs, I don’t really know how I could change permissions so those would be allowed. I also checked apt list --installed and in the docker image se-linux or apparmor are not installed.

Oh just for the check: is the whole path accessible for the user that runs gitlab?

yeah tried that too and didn’t seem to change anything