Gitlab runner k8s executor not mounting secret volume

Replace this template with your information

Describe your question in as much detail as possible:
Dear Gitlab community,

I have 2 local gitlab instances:

  • mygitlab.home
  • myothergitlab.home
    Then I have a k8s cluster with a gitlab runner installed using the official helm chart, I also have a secret volume configured to mount into the runner.

My problem is that when I push to mygitlab.home, the runner works as expected meaning it mounts the secret volume, however, if I push to myothergitlab.home, then the secret volume is not mounted into the gitlab runner.

What is the reason behind this?

imagePullPolicy: IfNotPresent
replicas: 1
gitlabUrl: https://mygitlab.home/
runnerRegistrationToken: "--REDACTED--"
terminationGracePeriodSeconds: 3600
concurrent: 15
checkInterval: 30
rbac:
  create: true
  rules:
    - resources: ["pods", "pods/exec", "secrets", "configmaps", "pods/attach"]
      verbs: ["get", "list", "watch", "create", "patch", "delete", "update"]

  clusterWideAccess: false

  ## Use podSecurity Policy
  ## ref: https://kubernetes.io/docs/concepts/policy/pod-security-policy/
  podSecurityPolicy:
    enabled: true
    resourceNames:
      - gitlab-runner

metrics:
  enabled: true

runners:
  name: "new-runner"
  locked: false
  config: |
    [[runners]]
      clone_url = "https://mygitlab.home./"
      [runners.kubernetes]
      privileged = false
      [[runners.kubernetes.volumes.secret]]
        name = "mysecrets"
        mount_path = "/mysecrets"
        read_only = true
        [runners.kubernetes.volumes.secret.items]
          "username" = "username"
          "password" = "password"
      [runners.kubernetes.volumes]
        [[runners.kubernetes.volumes.empty_dir]]
          name = "repo"
          mount_path = "/builds"
          medium = "Memory"
  executor: kubernetes
  tags: "test"

And this is my pipeline:

variables:
  GIT_SUBMODULE_STRATEGY: recursive
  VCS_GIT: myanotheranothergitlab.home/vcs/

stages:
  - pre
  - build
  - test

sync-git-branch:
  image: registry.gitlab.com/gitlab-org/gitlab-runner:alpine-v15.2.1
  stage: pre
  script:
    - USER=$(cat /mysecrets/username)
    - PASS=$(cat /mysecrets/password)
    - git config --local http.sslverify false
    - git remote add vcs https://${USER}:${PASS}@${VCS_GIT}/${CI_PROJECT_NAME}
    - git fetch origin
    - git fetch vcs
    - git submodule update --remote
    - git push vcs HEAD:refs/heads/${CI_COMMIT_BRANCH}
    - git push vcs HEAD:${CI_COMMIT_REF_NAME}
  rules:
    - if: $CI_COMMIT_BRANCH
  tags:
    - test
  • What are you seeing, and how does that differ from what you expect to see?

The pipeline fails because the secret is not mounted:

$ USER=$(cat /mysecrets/username)
cat: can't open '/mysecrets/username': No such file or directory
Cleaning up project directory and file based variables 00:00
ERROR: Job failed: command terminated with exit code 1
  • Consider including screenshots, error messages, and/or other helpful visuals

  • What version are you on? Are you using self-managed or GitLab.com?

    • GitLab (Hint: /help):
      myothergitlab.home is version 14.9.2
      mygitlab.home is version 15.11.3

    • Runner (Hint: /admin/runners):
      gitlab runner version is 15.2.1

  • Add the CI configuration from .gitlab-ci.yml and other configuration if relevant (e.g. docker-compose.yml)

  • What troubleshooting steps have you already taken? Can you link to any docs or other resources so we know where you have been?

I have modified the pipeline with a sleep 20m to keep the container runner running, then I describe the runner and I can verify the secrets volume is not getting mounted

Thanks for taking the time to be thorough in your request, it really helps! :blush:

GitLab Runner can be linked only to a single instance of GitLab. In other words, if you want GitLab Runner for myothergitlab.home you need to deploy another one which is registered to the other instance.

Hi,

According to the documentation, a gitlab runner can be linked to multiple gitlab instances.

From the docs Registering runners | GitLab

Runner registration is the process that links the runner with one or more GitLab instances.

I don’t understand your point, could you please elaborate?

Hi @masber1

GitLab Runner deployed using official Helm Chart is limited to single GitLab instance, which you specify using the gitlabUrl in the values when deploying it.
If you need Runner for 2nd instance you need to deploy another with gitlabUrl: myothergitlab.home

Since you have only 1 Runner registered to mygitlab.home you can’t see that runner in myothergitlab.home instance and hence it can’t run any jobs from that instance.

Hi,

ok, sorry I was not clear enough on this, I did register the runner manually to the second gitlab instance, therefore I can see the runner on both instances and also can run the runner when I push code to either gitlab instance… that is not my problem.
My problem (again) is that the secret volume is missing when I push against myothergitlab.home.

hope it is clean now :slight_smile:

Manually, like kubectl exec into the Pod and running gitlab-runner register? That’s not supported. It will get lost after your Pod rotates for whatever reason.

If you have manually registered the runner to another instance you also need to manually write the config file for it as well. You have not pasted the actual config.toml, if you made any changes, from the Pod so it’s not possible to help.

But then again, it’s pointless to manually adjust the config file, because any changes are lost on Pod rotate.

I see, you are right, the configuration changes I introduced by registering the runner to the second gitlab instance are transient, that is an issue I have to fix when maturing this setup to production, but for the time being, the configuration changes are still in place, because otherwise, the runner would not show as online in the myothergitlab.home instance…

In regards to the gitlab runner configuration… I think I did share it since it is included in the helm values file which I posted in my original message

so to my understanding the original question still remains :slight_smile:

I do have a running gitlab runner successfully registered to two different local gitlab instances and my problem is that the secrets volume is getting mounted when the runner is submitted by one of the gitlab instances (the one I registered manually) even though the runner starts the pipeline

The config you have posted applies only to the first (automatically) registered Runner when deployed.
Your config.toml most likely looks like this:

concurrent = 15
check_interval = 30

[[runners]]
  name = "new runner"
  url = https://mygitlab.home./
  token = ....
  executor = "kubernetes"
  clone_url = "https://mygitlab.home./"
  [runners.kubernetes]
    privileged = false
    [[runners.kubernetes.volumes.secret]]
        name = "mysecrets"
        mount_path = "/mysecrets"
        read_only = true
    [runners.kubernetes.volumes.secret.items]
          "username" = "username"
          "password" = "password"
    [runners.kubernetes.volumes]
        [[runners.kubernetes.volumes.empty_dir]]
          name = "repo"
          mount_path = "/builds"
          medium = "Memory"

[[runners]]
  name = "???"
  url = "https://myothergitlab.home"
  token = ...
  executor = "kubernetes"

where the first section applies to the automatically registered Runner and the second section applies to your manually registered Runner.

I really strongly, strongly, STRONGLY discourage of manually adjusting the setup configured by Helm Chart.
Just deploy a 2nd Runner on the same cluster the proper way. Why not?

Also if you are thinking about “adding” the second section to the helm values, that might work, might not work (most likely) or can stop working any time, because the chart officially supports only 1:1 setup.

I understand now

thank you very much for your patience and explanation, yes, the best solution is what you said and to deploy a new gitlab runner

thank you very much!

@balonik sorry… do you know what I need to change in the values.yaml in the gitlab helm chart to deploy a second gitlab runner in the same k8s namespace I have another gitlab runner already running?

If I change the values file, then the old gitlab runner gets deleted and the new one stays in place (even though the registration token, gitlab url and runner tags are different)

I also tried setting up the name in the runner section with no luck.

Does the helm chart support deploying multiple runners under the same namespace do I need to create a new namespace for each new runner I need to deploy?

thank you

@masber1 yes, you just need to give it a different name, example:

# helm syntax
# helm install <name> gitlab/gitlab-runner -f <path_to_values.yaml>
# install for mygitlab
helm install gitlab-runner-mygitlab gitlab/gitlab-runner -f values-mygitlab.yaml
# install for myothergitlab
helm install gitlab-runner-myothergitlab gitlab/gitlab-runner -f values-myothergitlab.yaml