Gitlab-runner and caching using minio in a subdir

GitLab CI/CD
1

I have a self-hosted instance of gitlab-ce:13.9.2-ce.0 with gitlab-runner:v13.9.0 running in a docker swarm. The latter’s configuration file is

gitlab-runner-config.toml 
concurrent = 1
check_interval = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "local"
  url = "http://gitlab:9080/git" # gitlab-ce is a subdir of the same subdomain
  token = "..token.."
  executor = "docker"
  cache_dir = "/cache"
  [runners.custom_build_dir]
  [runners.docker]
    tls_verify = false
    image = "debian"
    privileged = false
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/cache"]
    shm_size = 0
    extra_hosts = ["gitlab:172.19.0.2"]
    cache_dir = "/cache"
  [runners.cache]
    Type = "s3"
    Shared = true
    [runners.cache.s3]
      ServerAddress = "https://mysub.mydomain.com/minio/"
      AccessKey = "..akey.."
      SecretKey = "..skey.."
      BucketName = "gitlab-runner"
      Insecure = true

I was initially unable to get caching working, so I installed minio in the same swarm docker-compose.yml. Both instances are subdirectories of a subdomain behind traefik. The urls are:

Minio’s web interface seems to work just fine in this configuration. (And GitLab without this CI/CD issue has been working just fine.) CI/CD is relatively new to my setup.

docker-compose.yml 
version: '3.7'

services:

  traefik:
    image: traefik:1.7
    command: --api --docker --docker.exposedbydefault=false \
      --docker.domain=mysub.mydomain.com \
      --entryPoints="Name:https Address::443 TLS:/certs/mydomain-ucc.chained.crt,/certs/mydomain-ucc.key" \
      --entryPoints="Name:http Address::80 Redirect.EntryPoint:https" \
      --accessLog.format="json" \
      --accessLog.filePath="/log/access.log" \
      --traefikLog.format="json" \
      --traefikLog.filePath="/log/traefik.log" \
      --rest.entryPoint='traefik'
    ports:
      - target: 80
        published: 80
        mode: host
      - target: 443
        published: 443
        mode: host
      - "8081:8080" # webui
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /srv/docker/ssl:/certs:ro
      - /srv/docker/tk/logs:/log

  postgres:
    image: postgres:12.6
    ports:
      - "5432:5432"
    volumes:
      - /srv/docker/postgres/12/data:/var/lib/postgresql/data
      - /srv/docker/postgres/ssl:/ssl:ro
      - ./assets/postgres-initdb.d:/docker-entrypoint-initdb.d:ro
    secrets:
      - postgres_password

  redis:
    image: redis:5
    ports:
      - "6379:6379"

  gitlab:
    image: gitlab/gitlab-ce:13.9.2-ce.0
    depends_on:
      - postgres
      - redis
      - traefik
    environment:
      GITLAB_OMNIBUS_CONFIG: "from_file('/gitlab_configs.rb')"
    secrets:
      - source: gitlab_db_password
        target: db_password
      - source: gitlab_initial_root_password
        target: initial_root_password
      - source: searchagent_password
        target: ldap_servers_main_password
      - smtp_password
      - incoming_email_password
    ports:
      - "10022:22"
    volumes:
      - /srv/docker/gl/gitlab_configs.rb:/gitlab_configs.rb:ro
      - /srv/docker/gl/config:/etc/gitlab
      - /srv/docker/gl/logs:/var/log/gitlab
      - /srv/docker/gl/data:/var/opt/gitlab
    labels:
      - traefik.enable=true
      - traefik.frontend.entryPoints=https
      - traefik.frontend.headers.referrerPolicy=same-origin
      - traefik.frontend.headers.STSSeconds=315360000
      - traefik.glmain.port=9080 # https://stackoverflow.com/a/44142963
      - traefik.glmain.frontend.rule=PathPrefix:/git

  minio:
    image: minio/minio:latest
    depends_on:
      - traefik
    restart: unless-stopped
    command: "server /data"
    ports:
      - "9000:9000"
    volumes:
      - "/srv/docker/minio/data:/data"
    labels:
      - traefik.enable=true
      - traefik.frontend.entryPoints=https
      - traefik.frontend.rule=PathPrefix:/minio
      - traefik.frontend.headers.referrerPolicy=same-origin
      - traefik.frontend.headers.STSSeconds=315360000
    secrets:
      - source: minio_access_key
        target: access_key
      - source: minio_secret_key
        target: secret_key

  gitlab-runner:
    image: gitlab/gitlab-runner:v13.9.0
    depends_on:
      - gitlab
      - minio
    deploy:
      mode: replicated
      replicas: 2
    volumes:
      - /srv/docker/gl/gitlab-runner-config:/etc/gitlab-runner
      - /var/run/docker.sock:/var/run/docker.sock

When I run a CI pipe, in the log is

Restoring cache
Checking cache for default...
No URL provided, cache will not be downloaded from shared cache server. Instead a local version of cache will be extracted. 
Successfully extracted cache

When I look at the output from the gitlab-runner,

gitlab-runner_1  | ERROR: Could not create cache adapter               error=cache adapter could not be initialized: error while creating S3 cache storage client: Endpoint url cannot have fully qualified paths.

which leads me to the source in gitlab’s utils.go:

	if endpointURL.Path != "/" && endpointURL.Path != "" {
		return ErrInvalidArgument("Endpoint url cannot have fully qualified paths.")
	}

I don’t want to put the minio instance on its own subdomain, but this code suggests that S3-like object storage must always reside at the top-directory of a subdomain. Is there a way around this? If not, it’s apparent I don’t understand the implications of S3-compatible storage and why this is strictly required.

2

Hi @bill.evans
Minio S3 endpoints won’t work on subpath. Minio must be on /.
You cannot specify protocol in ServerAddress. Protocol is https by default unless you specify Insecure which switches to http. Insecure doesn’t mean to disable Certificate checks when using https and it is not possible to disable SSL verification.

[runners.cache]
    Type = "s3"
    Path = "runner/cache"
    Shared = true
    [runners.cache.s3]
      ServerAddress = "mysub.mydomain.com"
      AccessKey = "somekey"
      SecretKey = "somekey"
      BucketName = "cache-bucket"
      Insecure = false
3

Thank you for the reply, @balonik. The corrections on ServerAddress protocol and Insecure are well taken; the latter was an artifact from a previous config that I didn’t understand enough to remove, and your comment helped to clear that up.

The canonical reference the minio’s refusal to support sub-directories is the reference I hadn’t found, thank you for pointing it out. (I don’t think the issue is that the standard does not support sub-directories, as the signature creation specs say nothing about the intermediate path, just that it must be the same on ingress and internal routing. The issue is that minio chooses to not support it. I might be missing something, but regardless I will work with it.)

You added Path="runner/cache" to your suggested configuration, and changed the BucketName. I think that you are suggesting I organize the cache so that (for instance) the runners use runner/cache for the path leading up to the bucket cache-bucket, where the exact names here are not critical but the premise of organizing buckets under paths is the important part. Is that right? I’m new to S3 object storage in general, so I’m not familiar with best-practices in that regard. (In fact, right now the only use is the gitlab-runners, though I would expect expanding into other client-services in the future.)

4

@bill.evans I have never worked with S3 endpoints under sub-path so I cannot tell how well it is supported in other tools.
The Path and BucketName I have in my example are indeed just examples and you can choose whatever names you like. It was not my intention to change the ones you have already. The Path is like directory structure in case you use single Bucket for multiple things or you don’t want to have files under /. And BucketName is really just a name.

Bucket is a top level container, which contains files under paths. The actual path of any object in S3 would be BucketName/Path/some_file so in this example https://cache-bucket.s3-website.eu-west-1.amazonaws.com/runner/cache/some_file

5

Your first comment pushed me over-the-top and into a working “cache”.

Ultimately, I had to go with a new subdomain for the minio instance, since they’ve been explicit about not supporting sub-dir paths. I chose to not use Path=, instead relying on the BucketName= set to a UUID for the gitlab-runner cache (where a hierarchical organization does not seem critical).

Also, since this was deployed in a docker swarm, I was trying to use docker container names, thinking it would be easier to keep the network traffic within the docker network. While that might be nice on paper, I found I was dancing around SSL certs and hostnames. Ultimately, while it’s a little more network traffic, I’m sticking with “real world” names and real routing.

gitlab-runner-config.toml 
concurrent = 1
check_interval = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "local"
  url = "https://mysub.mydomain.com/git"
  token = "..token.."
  executor = "docker"
  cache_dir = "/cache"
  [runners.custom_build_dir]
  [runners.docker]
    tls_verify = false
    image = "debian"
    privileged = false
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/cache"]
    shm_size = 0
    cache_dir = "/cache"
  [runners.cache]
    Type = "s3"
    Shared = true
    [runners.cache.s3]
      ServerAddress = "myminio.mydomain.com"
      AccessKey = "..akey.."
      SecretKey = "..skey.."
      BucketName = "my-uuid-key"
      Insecure = false

Thank you!