"500 Internal Server Error" with job artifacts on minio

I’m running gitlab-ce on-prem with min.io as a local S3 service. CI/CD caching is working, and basic connectivity with the S3-compatible minio is good. (Versions: gitlab-ce:13.9.2-ce.0, gitlab-runner:v13.9.0, and minio/minio:latest currently c253244b6fb0.)

Is there additional configuration to differentiate between job-artifacts and pipeline-artifacts and storing them in on-prem S3-compatible object storage?

In my test repo, the “build” stage builds a sparse R package. When I was using local in-gitlab job artifacts, it succeeds and moves on to the “test” and “deploy” stages, no problems. (And that works with S3-stored cache, though that configuration is solely within gitlab-runner.) Now that I’ve configured minio as a local S3-compatible object storage, though, it fails.

...
Created cache
Uploading artifacts for successful job
Uploading artifacts...
/builds/git/mygroup/citest/ci/build/*.tar.gz: found 1 matching files and directories 
/builds/git/mygroup/citest/ci/lib: found 67 matching files and directories 
WARNING: Uploading artifacts as "archive" to coordinator... failed  id=397 responseStatus=500 Internal Server Error status=500 token=q42snHs9
WARNING: Retrying...                                context=artifacts-uploader error=invalid argument
WARNING: Uploading artifacts as "archive" to coordinator... failed  id=397 responseStatus=500 Internal Server Error status=500 token=q42snHs9
WARNING: Retrying...                                context=artifacts-uploader error=invalid argument
WARNING: Uploading artifacts as "archive" to coordinator... failed  id=397 responseStatus=500 Internal Server Error status=500 token=q42snHs9
FATAL: invalid argument                            
Cleaning up file based variables
ERROR: Job failed: exit code 1

The only fail is in the artifact-storage attempt. (With in-gitlab artifacts, this all succeeded and worked.)

I can find no reference of “invalid argument” within the minio logs (which say nothing during this time). I’m not certain that it is a minio-problem.

The relevant portion of the gitlab config:

gitlab_rails['object_store']['enabled'] = true
gitlab_rails['object_store']['proxy_download'] = false
gitlab_rails['object_store']['connection'] = {
  'provider' => 'AWS',
  'host' => "minio.mydomain.com",
  # 'region' => '',
  'aws_access_key_id' => '<AWS_ACCESS_KEY_ID>',
  'aws_secret_access_key' => '<AWS_SECRET_ACCESS_KEY>',
  'path_style' => true
}
gitlab_rails['object_store']['objects']['artifacts']['bucket'] = 'gitlab-artifacts-storage'
gitlab_rails['object_store']['objects']['external_diffs']['enabled'] = false
gitlab_rails['object_store']['objects']['lfs']['enabled'] = false
gitlab_rails['object_store']['objects']['uploads']['bucket'] = 'gitlab-uploads-storage'
gitlab_rails['object_store']['objects']['packages']['enabled'] = false
gitlab_rails['object_store']['objects']['dependency_proxy']['enabled'] = false
gitlab_rails['object_store']['objects']['terraform_state']['enabled'] = false
gitlab_rails['object_store']['objects']['pages']['enabled'] = false

That configuration is adapted from Object Storage | GitLab, deselecting storage of components I don’t think I need. I added 'path_style' => true because without it, the default of bucket.minio.mydomain.com was not resolved correctly (so this is mostly-S3-compatible, not perfectly AWS). I have also tried this with 'proxy_download' of true, no change.

I’m logged into the minio console, and the gitlab-artifacts-storage bucket exists. In fact, after these failed “build” tests, I’m seeing newly created job artifacts stored in this bucket (as job.log), so I know that basic connectivity (i.e., access key and secret) works:

image

In my research, I’ve seen similar errors linked to nginx rev-proxy issues, which suggests that something in the traefik configuration (or just its presence) might be an issue. It is merely passing traffic, it is doing no path-translation/stripping. Regardless, all of the rev-proxy discussion was on nginx, often attempting path-munging of some sort, and was resolved with some seemingly-unrelated change to the nginx configuration. I haven’t found any that map into the traefik domain.

traefik access.log

(Some HTTP 200 responses, some 404, I have not found a reason why.)

{"BackendAddr":"172.19.0.2:9000","BackendName":"backend-minio-myswarm","BackendURL":{"Scheme":"http","Opaque":"","User":null,"Host":"172.19.0.2:9000","Path":"","RawPath":"","ForceQuery":false,"RawQuery":"","Fragment":""},"ClientAddr":"172.19.0.1:44742","ClientHost":"172.19.0.1","ClientPort":"44742","ClientUsername":"-","DownstreamContentSize":329,"DownstreamStatus":200,"DownstreamStatusLine":"200 OK","Duration":3472517,"FrontendName":"Host-minio-mydomain-com-2","OriginContentSize":329,"OriginDuration":3422850,"OriginStatus":200,"OriginStatusLine":"200 OK","Overhead":49667,"RequestAddr":"minio.mydomain.com","RequestContentSize":0,"RequestCount":131470,"RequestHost":"minio.mydomain.com","RequestLine":"POST /gitlab-artifacts-storage/tmp/uploads/1616099873-1316-0002-0683-2dfe06e7a451447ff0b4a5518c8e19c6?uploads HTTP/1.1","RequestMethod":"POST","RequestPath":"/gitlab-artifacts-storage/tmp/uploads/1616099873-1316-0002-0683-2dfe06e7a451447ff0b4a5518c8e19c6?uploads","RequestPort":"-","RequestProtocol":"HTTP/1.1","RetryAttempts":0,"StartLocal":"2021-03-18T20:37:53.890788923Z","StartUTC":"2021-03-18T20:37:53.890788923Z","downstream_Accept-Ranges":"bytes","downstream_Content-Length":"329","downstream_Content-Security-Policy":"block-all-mixed-content","downstream_Content-Type":"application/xml","downstream_Date":"Thu, 18 Mar 2021 20:37:53 GMT","downstream_Referrer-Policy":"same-origin","downstream_Server":"MinIO","downstream_Strict-Transport-Security":"max-age=315360000","downstream_Vary":"Origin","downstream_X-Amz-Request-Id":"166D8A45D8DDC4B3","downstream_X-Xss-Protection":"1; mode=block","level":"info","msg":"","origin_Accept-Ranges":"bytes","origin_Content-Length":"329","origin_Content-Security-Policy":"block-all-mixed-content","origin_Content-Type":"application/xml","origin_Date":"Thu, 18 Mar 2021 20:37:53 GMT","origin_Referrer-Policy":"same-origin","origin_Server":"MinIO","origin_Strict-Transport-Security":"max-age=315360000","origin_Vary":"Origin","origin_X-Amz-Request-Id":"166D8A45D8DDC4B3","origin_X-Xss-Protection":"1; mode=block","request_Authorization":"AWS4-HMAC-SHA256 Credential=MyS3AccessKey/20210318//s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=SomeSignature","request_Content-Length":"0","request_User-Agent":"fog-core/2.1.0","request_X-Amz-Content-Sha256":"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855","request_X-Amz-Date":"20210318T203753Z","time":"2021-03-18T20:37:53Z"}
{"BackendAddr":"172.19.0.2:9000","BackendName":"backend-minio-myswarm","BackendURL":{"Scheme":"http","Opaque":"","User":null,"Host":"172.19.0.2:9000","Path":"","RawPath":"","ForceQuery":false,"RawQuery":"","Fragment":""},"ClientAddr":"172.19.0.1:44748","ClientHost":"172.19.0.1","ClientPort":"44748","ClientUsername":"-","DownstreamContentSize":467,"DownstreamStatus":404,"DownstreamStatusLine":"404 Not Found","Duration":1571234,"FrontendName":"Host-minio-mydomain-com-2","OriginContentSize":467,"OriginDuration":1525159,"OriginStatus":404,"OriginStatusLine":"404 Not Found","Overhead":46075,"RequestAddr":"minio.mydomain.com","RequestContentSize":0,"RequestCount":131472,"RequestHost":"minio.mydomain.com","RequestLine":"GET /gitlab-artifacts-storage/tmp/uploads/1616099873-1316-0002-0683-2dfe06e7a451447ff0b4a5518c8e19c6?X-Amz-Expires=15300\u0026X-Amz-Date=20210318T203753Z\u0026X-Amz-Algorithm=AWS4-HMAC-SHA256\u0026X-Amz-Credential=MyS3AccessKey%2F20210318%2F%2Fs3%2Faws4_request\u0026X-Amz-SignedHeaders=host\u0026X-Amz-Signature=SomeSignature HTTP/1.1","RequestMethod":"GET","RequestPath":"/gitlab-artifacts-storage/tmp/uploads/1616099873-1316-0002-0683-2dfe06e7a451447ff0b4a5518c8e19c6?X-Amz-Expires=15300\u0026X-Amz-Date=20210318T203753Z\u0026X-Amz-Algorithm=AWS4-HMAC-SHA256\u0026X-Amz-Credential=MyS3AccessKey%2F20210318%2F%2Fs3%2Faws4_request\u0026X-Amz-SignedHeaders=host\u0026X-Amz-Signature=SomeSignature","RequestPort":"-","RequestProtocol":"HTTP/1.1","RetryAttempts":0,"StartLocal":"2021-03-18T20:37:54.090024071Z","StartUTC":"2021-03-18T20:37:54.090024071Z","downstream_Accept-Ranges":"bytes","downstream_Content-Length":"467","downstream_Content-Security-Policy":"block-all-mixed-content","downstream_Content-Type":"application/xml","downstream_Date":"Thu, 18 Mar 2021 20:37:54 GMT","downstream_Referrer-Policy":"same-origin","downstream_Server":"MinIO","downstream_Strict-Transport-Security":"max-age=315360000","downstream_Vary":"Origin","downstream_X-Amz-Request-Id":"166D8A45E4BD6115","downstream_X-Xss-Protection":"1; mode=block","level":"info","msg":"","origin_Accept-Ranges":"bytes","origin_Content-Length":"467","origin_Content-Security-Policy":"block-all-mixed-content","origin_Content-Type":"application/xml","origin_Date":"Thu, 18 Mar 2021 20:37:54 GMT","origin_Referrer-Policy":"same-origin","origin_Server":"MinIO","origin_Strict-Transport-Security":"max-age=315360000","origin_Vary":"Origin","origin_X-Amz-Request-Id":"166D8A45E4BD6115","origin_X-Xss-Protection":"1; mode=block","request_User-Agent":"Go-http-client/1.1","time":"2021-03-18T20:37:54Z"}
{"BackendAddr":"172.19.0.2:9000","BackendName":"backend-minio-myswarm","BackendURL":{"Scheme":"http","Opaque":"","User":null,"Host":"172.19.0.2:9000","Path":"","RawPath":"","ForceQuery":false,"RawQuery":"","Fragment":""},"ClientAddr":"172.19.0.1:44754","ClientHost":"172.19.0.1","ClientPort":"44754","ClientUsername":"-","DownstreamContentSize":0,"DownstreamStatus":404,"DownstreamStatusLine":"404 Not Found","Duration":1205368,"FrontendName":"Host-minio-mydomain-com-2","OriginContentSize":0,"OriginDuration":1145885,"OriginStatus":404,"OriginStatusLine":"404 Not Found","Overhead":59483,"RequestAddr":"minio.mydomain.com","RequestContentSize":0,"RequestCount":131473,"RequestHost":"minio.mydomain.com","RequestLine":"HEAD /gitlab-artifacts-storage/tmp/uploads/1616099873-1316-0002-0683-2dfe06e7a451447ff0b4a5518c8e19c6 HTTP/1.1","RequestMethod":"HEAD","RequestPath":"/gitlab-artifacts-storage/tmp/uploads/1616099873-1316-0002-0683-2dfe06e7a451447ff0b4a5518c8e19c6","RequestPort":"-","RequestProtocol":"HTTP/1.1","RetryAttempts":0,"StartLocal":"2021-03-18T20:37:54.162829022Z","StartUTC":"2021-03-18T20:37:54.162829022Z","downstream_Accept-Ranges":"bytes","downstream_Content-Length":"0","downstream_Content-Security-Policy":"block-all-mixed-content","downstream_Date":"Thu, 18 Mar 2021 20:37:54 GMT","downstream_Referrer-Policy":"same-origin","downstream_Server":"MinIO","downstream_Strict-Transport-Security":"max-age=315360000","downstream_Vary":"Origin","downstream_X-Amz-Request-Id":"166D8A45E9114D33","downstream_X-Xss-Protection":"1; mode=block","level":"info","msg":"","origin_Accept-Ranges":"bytes","origin_Content-Length":"0","origin_Content-Security-Policy":"block-all-mixed-content","origin_Date":"Thu, 18 Mar 2021 20:37:54 GMT","origin_Referrer-Policy":"same-origin","origin_Server":"MinIO","origin_Strict-Transport-Security":"max-age=315360000","origin_Vary":"Origin","origin_X-Amz-Request-Id":"166D8A45E9114D33","origin_X-Xss-Protection":"1; mode=block","request_Authorization":"AWS4-HMAC-SHA256 Credential=MyS3AccessKey/20210318//s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=SomeSignature","request_Content-Length":"0","request_User-Agent":"fog-core/2.1.0","request_X-Amz-Content-Sha256":"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855","request_X-Amz-Date":"20210318T203754Z","time":"2021-03-18T20:37:54Z"}
{"BackendAddr":"172.19.0.2:9000","BackendName":"backend-minio-myswarm","BackendURL":{"Scheme":"http","Opaque":"","User":null,"Host":"172.19.0.2:9000","Path":"","RawPath":"","ForceQuery":false,"RawQuery":"","Fragment":""},"ClientAddr":"172.19.0.1:44758","ClientHost":"172.19.0.1","ClientPort":"44758","ClientUsername":"-","DownstreamContentSize":0,"DownstreamStatus":404,"DownstreamStatusLine":"404 Not Found","Duration":1087332,"FrontendName":"Host-minio-mydomain-com-2","OriginContentSize":0,"OriginDuration":1031618,"OriginStatus":404,"OriginStatusLine":"404 Not Found","Overhead":55714,"RequestAddr":"minio.mydomain.com","RequestContentSize":0,"RequestCount":131474,"RequestHost":"minio.mydomain.com","RequestLine":"HEAD /gitlab-artifacts-storage/tmp/uploads/1616099873-1316-0002-0683-2dfe06e7a451447ff0b4a5518c8e19c6 HTTP/1.1","RequestMethod":"HEAD","RequestPath":"/gitlab-artifacts-storage/tmp/uploads/1616099873-1316-0002-0683-2dfe06e7a451447ff0b4a5518c8e19c6","RequestPort":"-","RequestProtocol":"HTTP/1.1","RetryAttempts":0,"StartLocal":"2021-03-18T20:37:54.206750001Z","StartUTC":"2021-03-18T20:37:54.206750001Z","downstream_Accept-Ranges":"bytes","downstream_Content-Length":"0","downstream_Content-Security-Policy":"block-all-mixed-content","downstream_Date":"Thu, 18 Mar 2021 20:37:54 GMT","downstream_Referrer-Policy":"same-origin","downstream_Server":"MinIO","downstream_Strict-Transport-Security":"max-age=315360000","downstream_Vary":"Origin","downstream_X-Amz-Request-Id":"166D8A45EBAE7A4E","downstream_X-Xss-Protection":"1; mode=block","level":"info","msg":"","origin_Accept-Ranges":"bytes","origin_Content-Length":"0","origin_Content-Security-Policy":"block-all-mixed-content","origin_Date":"Thu, 18 Mar 2021 20:37:54 GMT","origin_Referrer-Policy":"same-origin","origin_Server":"MinIO","origin_Strict-Transport-Security":"max-age=315360000","origin_Vary":"Origin","origin_X-Amz-Request-Id":"166D8A45EBAE7A4E","origin_X-Xss-Protection":"1; mode=block","request_Authorization":"AWS4-HMAC-SHA256 Credential=MyS3AccessKey/20210318//s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=SomeSignature","request_Content-Length":"0","request_User-Agent":"fog-core/2.1.0","request_X-Amz-Content-Sha256":"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855","request_X-Amz-Date":"20210318T203754Z","time":"2021-03-18T20:37:54Z"}
{"BackendAddr":"172.19.0.2:9000","BackendName":"backend-minio-myswarm","BackendURL":{"Scheme":"http","Opaque":"","User":null,"Host":"172.19.0.2:9000","Path":"","RawPath":"","ForceQuery":false,"RawQuery":"","Fragment":""},"ClientAddr":"172.19.0.1:44762","ClientHost":"172.19.0.1","ClientPort":"44762","ClientUsername":"-","DownstreamContentSize":0,"DownstreamStatus":404,"DownstreamStatusLine":"404 Not Found","Duration":1126408,"FrontendName":"Host-minio-mydomain-com-2","OriginContentSize":0,"OriginDuration":1068170,"OriginStatus":404,"OriginStatusLine":"404 Not Found","Overhead":58238,"RequestAddr":"minio.mydomain.com","RequestContentSize":0,"RequestCount":131475,"RequestHost":"minio.mydomain.com","RequestLine":"HEAD /gitlab-artifacts-storage/tmp/uploads/1616099873-1316-0002-0683-2dfe06e7a451447ff0b4a5518c8e19c6 HTTP/1.1","RequestMethod":"HEAD","RequestPath":"/gitlab-artifacts-storage/tmp/uploads/1616099873-1316-0002-0683-2dfe06e7a451447ff0b4a5518c8e19c6","RequestPort":"-","RequestProtocol":"HTTP/1.1","RetryAttempts":0,"StartLocal":"2021-03-18T20:37:54.243629562Z","StartUTC":"2021-03-18T20:37:54.243629562Z","downstream_Accept-Ranges":"bytes","downstream_Content-Length":"0","downstream_Content-Security-Policy":"block-all-mixed-content","downstream_Date":"Thu, 18 Mar 2021 20:37:54 GMT","downstream_Referrer-Policy":"same-origin","downstream_Server":"MinIO","downstream_Strict-Transport-Security":"max-age=315360000","downstream_Vary":"Origin","downstream_X-Amz-Request-Id":"166D8A45EDE0062E","downstream_X-Xss-Protection":"1; mode=block","level":"info","msg":"","origin_Accept-Ranges":"bytes","origin_Content-Length":"0","origin_Content-Security-Policy":"block-all-mixed-content","origin_Date":"Thu, 18 Mar 2021 20:37:54 GMT","origin_Referrer-Policy":"same-origin","origin_Server":"MinIO","origin_Strict-Transport-Security":"max-age=315360000","origin_Vary":"Origin","origin_X-Amz-Request-Id":"166D8A45EDE0062E","origin_X-Xss-Protection":"1; mode=block","request_Authorization":"AWS4-HMAC-SHA256 Credential=MyS3AccessKey/20210318//s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=SomeSignature","request_Content-Length":"0","request_User-Agent":"fog-core/2.1.0","request_X-Amz-Content-Sha256":"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855","request_X-Amz-Date":"20210318T203754Z","time":"2021-03-18T20:37:54Z"}
{"BackendAddr":"172.19.0.2:9000","BackendName":"backend-minio-myswarm","BackendURL":{"Scheme":"http","Opaque":"","User":null,"Host":"172.19.0.2:9000","Path":"","RawPath":"","ForceQuery":false,"RawQuery":"","Fragment":""},"ClientAddr":"172.19.0.1:44766","ClientHost":"172.19.0.1","ClientPort":"44766","ClientUsername":"-","DownstreamContentSize":0,"DownstreamStatus":404,"DownstreamStatusLine":"404 Not Found","Duration":1279861,"FrontendName":"Host-minio-mydomain-com-2","OriginContentSize":0,"OriginDuration":1227773,"OriginStatus":404,"OriginStatusLine":"404 Not Found","Overhead":52088,"RequestAddr":"minio.mydomain.com","RequestContentSize":0,"RequestCount":131476,"RequestHost":"minio.mydomain.com","RequestLine":"HEAD /gitlab-artifacts-storage/tmp/uploads/1616099873-1316-0002-0683-2dfe06e7a451447ff0b4a5518c8e19c6 HTTP/1.1","RequestMethod":"HEAD","RequestPath":"/gitlab-artifacts-storage/tmp/uploads/1616099873-1316-0002-0683-2dfe06e7a451447ff0b4a5518c8e19c6","RequestPort":"-","RequestProtocol":"HTTP/1.1","RetryAttempts":0,"StartLocal":"2021-03-18T20:37:54.277572935Z","StartUTC":"2021-03-18T20:37:54.277572935Z","downstream_Accept-Ranges":"bytes","downstream_Content-Length":"0","downstream_Content-Security-Policy":"block-all-mixed-content","downstream_Date":"Thu, 18 Mar 2021 20:37:54 GMT","downstream_Referrer-Policy":"same-origin","downstream_Server":"MinIO","downstream_Strict-Transport-Security":"max-age=315360000","downstream_Vary":"Origin","downstream_X-Amz-Request-Id":"166D8A45EFE57F90","downstream_X-Xss-Protection":"1; mode=block","level":"info","msg":"","origin_Accept-Ranges":"bytes","origin_Content-Length":"0","origin_Content-Security-Policy":"block-all-mixed-content","origin_Date":"Thu, 18 Mar 2021 20:37:54 GMT","origin_Referrer-Policy":"same-origin","origin_Server":"MinIO","origin_Strict-Transport-Security":"max-age=315360000","origin_Vary":"Origin","origin_X-Amz-Request-Id":"166D8A45EFE57F90","origin_X-Xss-Protection":"1; mode=block","request_Authorization":"AWS4-HMAC-SHA256 Credential=MyS3AccessKey/20210318//s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=SomeSignature","request_Content-Length":"0","request_User-Agent":"fog-core/2.1.0","request_X-Amz-Content-Sha256":"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855","request_X-Amz-Date":"20210318T203754Z","time":"2021-03-18T20:37:54Z"}
{"BackendAddr":"172.19.0.2:9000","BackendName":"backend-minio-myswarm","BackendURL":{"Scheme":"http","Opaque":"","User":null,"Host":"172.19.0.2:9000","Path":"","RawPath":"","ForceQuery":false,"RawQuery":"","Fragment":""},"ClientAddr":"172.19.0.1:44774","ClientHost":"172.19.0.1","ClientPort":"44774","ClientUsername":"-","DownstreamContentSize":329,"DownstreamStatus":200,"DownstreamStatusLine":"200 OK","Duration":4317260,"FrontendName":"Host-minio-mydomain-com-2","OriginContentSize":329,"OriginDuration":4261951,"OriginStatus":200,"OriginStatusLine":"200 OK","Overhead":55309,"RequestAddr":"minio.mydomain.com","RequestContentSize":0,"RequestCount":131480,"RequestHost":"minio.mydomain.com","RequestLine":"POST /gitlab-artifacts-storage/tmp/uploads/1616099875-1422-0002-1921-4427532294f444d4582d12ec3b75ba3a?uploads HTTP/1.1","RequestMethod":"POST","RequestPath":"/gitlab-artifacts-storage/tmp/uploads/1616099875-1422-0002-1921-4427532294f444d4582d12ec3b75ba3a?uploads","RequestPort":"-","RequestProtocol":"HTTP/1.1","RetryAttempts":0,"StartLocal":"2021-03-18T20:37:55.503920248Z","StartUTC":"2021-03-18T20:37:55.503920248Z","downstream_Accept-Ranges":"bytes","downstream_Content-Length":"329","downstream_Content-Security-Policy":"block-all-mixed-content","downstream_Content-Type":"application/xml","downstream_Date":"Thu, 18 Mar 2021 20:37:55 GMT","downstream_Referrer-Policy":"same-origin","downstream_Server":"MinIO","downstream_Strict-Transport-Security":"max-age=315360000","downstream_Vary":"Origin","downstream_X-Amz-Request-Id":"166D8A4639004631","downstream_X-Xss-Protection":"1; mode=block","level":"info","msg":"","origin_Accept-Ranges":"bytes","origin_Content-Length":"329","origin_Content-Security-Policy":"block-all-mixed-content","origin_Content-Type":"application/xml","origin_Date":"Thu, 18 Mar 2021 20:37:55 GMT","origin_Referrer-Policy":"same-origin","origin_Server":"MinIO","origin_Strict-Transport-Security":"max-age=315360000","origin_Vary":"Origin","origin_X-Amz-Request-Id":"166D8A4639004631","origin_X-Xss-Protection":"1; mode=block","request_Authorization":"AWS4-HMAC-SHA256 Credential=MyS3AccessKey/20210318//s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=SomeSignature","request_Content-Length":"0","request_User-Agent":"fog-core/2.1.0","request_X-Amz-Content-Sha256":"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855","request_X-Amz-Date":"20210318T203755Z","time":"2021-03-18T20:37:55Z"}
{"BackendAddr":"172.19.0.2:9000","BackendName":"backend-minio-myswarm","BackendURL":{"Scheme":"http","Opaque":"","User":null,"Host":"172.19.0.2:9000","Path":"","RawPath":"","ForceQuery":false,"RawQuery":"","Fragment":""},"ClientAddr":"172.19.0.1:44778","ClientHost":"172.19.0.1","ClientPort":"44778","ClientUsername":"-","DownstreamContentSize":467,"DownstreamStatus":404,"DownstreamStatusLine":"404 Not Found","Duration":1879630,"FrontendName":"Host-minio-mydomain-com-2","OriginContentSize":467,"OriginDuration":1830988,"OriginStatus":404,"OriginStatusLine":"404 Not Found","Overhead":48642,"RequestAddr":"minio.mydomain.com","RequestContentSize":0,"RequestCount":131482,"RequestHost":"minio.mydomain.com","RequestLine":"GET /gitlab-artifacts-storage/tmp/uploads/1616099875-1422-0002-1921-4427532294f444d4582d12ec3b75ba3a?X-Amz-Expires=15300\u0026X-Amz-Date=20210318T203755Z\u0026X-Amz-Algorithm=AWS4-HMAC-SHA256\u0026X-Amz-Credential=MyS3AccessKey%2F20210318%2F%2Fs3%2Faws4_request\u0026X-Amz-SignedHeaders=host\u0026X-Amz-Signature=SomeSignature HTTP/1.1","RequestMethod":"GET","RequestPath":"/gitlab-artifacts-storage/tmp/uploads/1616099875-1422-0002-1921-4427532294f444d4582d12ec3b75ba3a?X-Amz-Expires=15300\u0026X-Amz-Date=20210318T203755Z\u0026X-Amz-Algorithm=AWS4-HMAC-SHA256\u0026X-Amz-Credential=MyS3AccessKey%2F20210318%2F%2Fs3%2Faws4_request\u0026X-Amz-SignedHeaders=host\u0026X-Amz-Signature=SomeSignature","RequestPort":"-","RequestProtocol":"HTTP/1.1","RetryAttempts":0,"StartLocal":"2021-03-18T20:37:55.703174362Z","StartUTC":"2021-03-18T20:37:55.703174362Z","downstream_Accept-Ranges":"bytes","downstream_Content-Length":"467","downstream_Content-Security-Policy":"block-all-mixed-content","downstream_Content-Type":"application/xml","downstream_Date":"Thu, 18 Mar 2021 20:37:55 GMT","downstream_Referrer-Policy":"same-origin","downstream_Server":"MinIO","downstream_Strict-Transport-Security":"max-age=315360000","downstream_Vary":"Origin","downstream_X-Amz-Request-Id":"166D8A4644E482E7","downstream_X-Xss-Protection":"1; mode=block","level":"info","msg":"","origin_Accept-Ranges":"bytes","origin_Content-Length":"467","origin_Content-Security-Policy":"block-all-mixed-content","origin_Content-Type":"application/xml","origin_Date":"Thu, 18 Mar 2021 20:37:55 GMT","origin_Referrer-Policy":"same-origin","origin_Server":"MinIO","origin_Strict-Transport-Security":"max-age=315360000","origin_Vary":"Origin","origin_X-Amz-Request-Id":"166D8A4644E482E7","origin_X-Xss-Protection":"1; mode=block","request_User-Agent":"Go-http-client/1.1","time":"2021-03-18T20:37:55Z"}
{"BackendAddr":"172.19.0.2:9000","BackendName":"backend-minio-myswarm","BackendURL":{"Scheme":"http","Opaque":"","User":null,"Host":"172.19.0.2:9000","Path":"","RawPath":"","ForceQuery":false,"RawQuery":"","Fragment":""},"ClientAddr":"172.19.0.1:44782","ClientHost":"172.19.0.1","ClientPort":"44782","ClientUsername":"-","DownstreamContentSize":0,"DownstreamStatus":404,"DownstreamStatusLine":"404 Not Found","Duration":2076600,"FrontendName":"Host-minio-mydomain-com-2","OriginContentSize":0,"OriginDuration":2009920,"OriginStatus":404,"OriginStatusLine":"404 Not Found","Overhead":66680,"RequestAddr":"minio.mydomain.com","RequestContentSize":0,"RequestCount":131484,"RequestHost":"minio.mydomain.com","RequestLine":"HEAD /gitlab-artifacts-storage/tmp/uploads/1616099875-1422-0002-1921-4427532294f444d4582d12ec3b75ba3a HTTP/1.1","RequestMethod":"HEAD","RequestPath":"/gitlab-artifacts-storage/tmp/uploads/1616099875-1422-0002-1921-4427532294f444d4582d12ec3b75ba3a","RequestPort":"-","RequestProtocol":"HTTP/1.1","RetryAttempts":0,"StartLocal":"2021-03-18T20:37:56.652353505Z","StartUTC":"2021-03-18T20:37:56.652353505Z","downstream_Accept-Ranges":"bytes","downstream_Content-Length":"0","downstream_Content-Security-Policy":"block-all-mixed-content","downstream_Date":"Thu, 18 Mar 2021 20:37:56 GMT","downstream_Referrer-Policy":"same-origin","downstream_Server":"MinIO","downstream_Strict-Transport-Security":"max-age=315360000","downstream_Vary":"Origin","downstream_X-Amz-Request-Id":"166D8A467D7A9332","downstream_X-Xss-Protection":"1; mode=block","level":"info","msg":"","origin_Accept-Ranges":"bytes","origin_Content-Length":"0","origin_Content-Security-Policy":"block-all-mixed-content","origin_Date":"Thu, 18 Mar 2021 20:37:56 GMT","origin_Referrer-Policy":"same-origin","origin_Server":"MinIO","origin_Strict-Transport-Security":"max-age=315360000","origin_Vary":"Origin","origin_X-Amz-Request-Id":"166D8A467D7A9332","origin_X-Xss-Protection":"1; mode=block","request_Authorization":"AWS4-HMAC-SHA256 Credential=MyS3AccessKey/20210318//s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=SomeSignature","request_Content-Length":"0","request_User-Agent":"fog-core/2.1.0","request_X-Amz-Content-Sha256":"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855","request_X-Amz-Date":"20210318T203756Z","time":"2021-03-18T20:37:56Z"}

I understand there is a difference between job artifacts and pipeline artifacts, so I think what I’m missing is a configuration distinction between “job” and “pipeline”.

.gitlab-ci.yml snippet
variables:
  GIT_DEPTH: 10
  R_LIBS_USER: "$CI_PROJECT_DIR/ci/lib"
  BUILD_DIR: "$CI_PROJECT_DIR/ci/build"
  CHECK_DIR: "$CI_PROJECT_DIR/ci/logs"
  BUILD_LOGS_DIR: "$CI_PROJECT_DIR/ci/logs/$CI_PROJECT_NAME.Rcheck"

default:
  image: rocker/shiny-verse:4.0.3
  interruptible: true

build-package:
  stage: build
  script:
    - mkdir -p "$R_LIBS_USER" "$BUILD_DIR"
    - R -e '
        devtools::install_deps(dependencies = TRUE, lib = Sys.getenv("R_LIBS_USER")) ;
        devtools::build(path = Sys.getenv("BUILD_DIR")) ;'
  artifacts:
    paths:
      - $BUILD_DIR/*.tar.gz
      - $R_LIBS_USER
  cache:
    key: "${CI_COMMIT_REF_SLUG}__cilib"
    paths:
      - $R_LIBS_USER

(Yes, it is both cached and an artifact. This is an artifact of my testing of the CI subsystem, not something I intend to maintain.)

This is in a docker-swarm behind a traefik reverse-proxy (and SSL terminator).

docker-compose.yml
version: '3.7'

services:

  traefik:
    image: traefik:1.7
    command: --api --docker --docker.exposedbydefault=false \
      --docker.domain=mysub.mydomain.com \
      --entryPoints="Name:https Address::443 TLS:/certs/mydomain-ucc.chained.crt,/certs/mydomain-ucc.key" \
      --entryPoints="Name:http Address::80 Redirect.EntryPoint:https" \
      --accessLog.format="json" \
      --accessLog.filePath="/log/access.log" \
      --traefikLog.format="json" \
      --traefikLog.filePath="/log/traefik.log" \
      --rest.entryPoint='traefik'
    ports:
      - target: 80
        published: 80
        mode: host
      - target: 443
        published: 443
        mode: host
      - "8081:8080" # webui
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /srv/docker/ssl:/certs:ro
      - /srv/docker/tk/logs:/log

  postgres:
    image: postgres:12.6
    ports:
      - "5432:5432"
    volumes:
      - /srv/docker/postgres/12/data:/var/lib/postgresql/data
      - /srv/docker/postgres/ssl:/ssl:ro
      - ./assets/postgres-initdb.d:/docker-entrypoint-initdb.d:ro
    secrets:
      - postgres_password

  redis:
    image: redis:5
    ports:
      - "6379:6379"

  gitlab:
    image: gitlab/gitlab-ce:13.9.4-ce.0
    depends_on:
      - postgres
      - redis
      - traefik
    environment:
      GITLAB_OMNIBUS_CONFIG: "from_file('/gitlab_configs.rb')"
    secrets:
      - source: gitlab_db_password
        target: db_password
      - source: gitlab_initial_root_password
        target: initial_root_password
      - source: searchagent_password
        target: ldap_servers_main_password
      - smtp_password
      - incoming_email_password
    ports:
      - "10022:22"
    volumes:
      - /srv/docker/gl/gitlab_configs.rb:/gitlab_configs.rb:ro
      - /srv/docker/gl/config:/etc/gitlab
      - /srv/docker/gl/logs:/var/log/gitlab
      - /srv/docker/gl/data:/var/opt/gitlab
    labels:
      - traefik.enable=true
      - traefik.frontend.entryPoints=https
      - traefik.frontend.headers.referrerPolicy=same-origin
      - traefik.frontend.headers.STSSeconds=315360000
      - traefik.glmain.port=9080 # https://stackoverflow.com/a/44142963
      - traefik.glmain.frontend.rule=PathPrefix:/git

  minio:
    image: minio/minio:latest
    depends_on:
      - traefik
    restart: unless-stopped
    command: "server /data"
    ports:
      - "9000:9000"
    volumes:
      - "/srv/docker/minio/data:/data"
    labels:
      - traefik.enable=true
      - traefik.frontend.entryPoints=https
      - traefik.frontend.rule=Host:mysub.mydomain.com
      - traefik.frontend.headers.referrerPolicy=same-origin
      - traefik.frontend.headers.STSSeconds=315360000
    secrets:
      - source: minio_access_key
        target: access_key
      - source: minio_secret_key
        target: secret_key

  gitlab-runner:
    image: gitlab/gitlab-runner:v13.9.0
    depends_on:
      - gitlab
      - minio
    deploy:
      mode: replicated
      replicas: 2
    volumes:
      - /srv/docker/gl/gitlab-runner-config:/etc/gitlab-runner
      - /var/run/docker.sock:/var/run/docker.sock

Got 500 error with ceph and minio, help me gitlab#297227 (closed)