Helm chart deployed gitlab-runner intermittently seems to fail to mount secret as volume

Problem to solve

Describe your question in as much detail as possible:
If I set ‘concurrent: 1’, no issues occur. But when I comment that out I occasionally see issues.

• What are you seeing, and how does that differ from what you expect to see?
  The error I see is that I’m unable to perform a ‘docker login’ to my onprem harbor server because of invalid certificates. This is the error seen if I do not provide a ca-bundle.

Steps to reproduce

  # configure runners
  runners:
    config: |
      [[runners]]
        [runners.kubernetes]
          namespace = "gitlab-runner"
          image = "ubuntu:22.04"
          privileged = true
        [[runners.kubernetes.volumes.secret]]
          name = "k-home-net"
          mount_path = "/etc/ssl/certs"
        [[runners.kubernetes.volumes.pvc]]
          name = "docker-certs"
          mount_path = "/certs/client"

Configuration

The kubernetes secret ‘k-home-net’ contains the ca-bundle which needs to be mounted in the runner so the runner can have access to onprem servers. It’s acting as if a secret can only be mounted in a single pod at a time.

Versions

latest helm chart

• GitLab.com SaaS
• Self-hosted Runners

Looks like secrets and configmaps can only be mounted by one pod at a time … created a job to copy the certs from the secret over to the pvc and mounted a rwx pvc instead.

apiVersion: batch/v1
kind: Job
metadata:
  name: copy-ca-bundle
spec:
  template:
    spec:
      restartPolicy: Never
      containers:
      - name: get-ca-bundle
        image: alpine/curl:8.9.1
        imagePullPolicy: Always
        command: ['sh', '-c']
        args:
        - |
          # copy certs from secret to pvc
          cp --verbose /etc/ssl/certs/ca.crt /tmp/certs/
          cat /tmp/certs/ca.crt

        volumeMounts:
        - name: ssl-certs
          mountPath: "/tmp/certs"
        - name: k-home-net
          mountPath: "/etc/ssl/certs"

      volumes:
      - name: ssl-certs
        persistentVolumeClaim:
          claimName: ssl-certs
      - name: k-home-net
        secret:
          secretName: k-home-net

I’m still seeing the same issue with a RWX cephfs pvc …

        [[runners.kubernetes.volumes.pvc]]
          name = "ssl-certs"
          mount_path = "/etc/ssl/certs"
        [[runners.kubernetes.volumes.pvc]]
          name = "docker-certs"
          mount_path = "/certs/client"

I tried with pvcs mapping an nfs share and still running into the same issue.

NFS for sure should have worked. What’s going on here?

image870×361 28.3 KB

i went back to using cephfs, i’ve been assured it works as rwx

here are 4 builds run at the same time, only 1 worked, its as if the pvc could only be mounted in one, even though it is RWX

image908×243 25.1 KB

image880×232 25.8 KB