Can't register gitlab runner within Kubernetes Cluster Manager

Dear GitLab Community,

im fairly new to GItLab, working on it for a week now and I’m currently stuck with applying GitLab Runner within my BareMetal Kubernetes Cluster and I search your assistance to help me with this Issue.

tl;dr: GitLab runner cant register on my GitLab due invalid certificate authority. x509: certificate is valid for ingress.local as I use a Private Certificate

My Environment:
Kubernetes v1.18.3 - 2 Master, 3 Nodes
Helm v3.3.4
GitLab 14.3.2-ee

My Issue:
I use the GitLab Kubernetes Cluster Management Project, from it I deploy Ingress - Gitlab Runner - Prometehus and MetalLB. The CI/CD Pipe runs through and deploys everything within the suggested Namespace gitlab-managed-apps.

kubectl get all -n gitlab-managed-apps:

NAME                                                         READY   STATUS             RESTARTS   AGE
pod/ingress-nginx-ingress-controller-6cbf95f5d4-mvxdt        1/1     Running            0          3d20h
pod/ingress-nginx-ingress-default-backend-77d64745d9-n75jj   1/1     Running            0          3d20h
pod/metallb-controller-5c57bcd798-r6ctr                      1/1     Running            0          3d18h
pod/metallb-speaker-fmphd                                    1/1     Running            0          3d18h
pod/metallb-speaker-fwqgc                                    1/1     Running            0          3d18h
pod/metallb-speaker-fz4mm                                    1/1     Running            0          3d18h
pod/metallb-speaker-p6xzp                                    1/1     Running            0          3d18h
pod/metallb-speaker-z2zvz                                    1/1     Running            0          3d18h
pod/prometheus-kube-state-metrics-7596c4bc64-hxk6n           1/1     Running            0          3d20h
pod/prometheus-prometheus-server-597dc9d9c7-hq5r4            2/2     Running            0          3d20h
pod/runner-gitlab-runner-5449984dfd-fmnhl                    0/1     CrashLoopBackOff   16         103m

NAME                                            TYPE           CLUSTER-IP      EXTERNAL-IP    PORT(S)                      AGE
service/ingress-nginx-ingress-controller        LoadBalancer   80:31695/TCP,443:30164/TCP   3d20h
service/ingress-nginx-ingress-default-backend   ClusterIP    <none>         80/TCP                       3d20h
service/prometheus-kube-state-metrics           ClusterIP      None            <none>         80/TCP,81/TCP                3d20h
service/prometheus-prometheus-server            ClusterIP   <none>         80/TCP                       3d20h

NAME                             DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
daemonset.apps/metallb-speaker   5         5         5       5            5    3d18h

NAME                                                    READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/ingress-nginx-ingress-controller        1/1     1            1           3d20h
deployment.apps/ingress-nginx-ingress-default-backend   1/1     1            1           3d20h
deployment.apps/metallb-controller                      1/1     1            1           3d18h
deployment.apps/prometheus-kube-state-metrics           1/1     1            1           3d20h
deployment.apps/prometheus-prometheus-server            1/1     1            1           3d20h
deployment.apps/runner-gitlab-runner                    0/1     1            0           3d20h

my GitLab Runner Pod crashes the whole time with following log output:

kubectl logs -n gitlab-managed-apps runner-gitlab-runner-5449984dfd-fmnhl

ERROR: Registering runner... failed                 runner=PJ-sm8tH status=couldn't execute POST against https://gitlab.*****.***/api/v4/runners: Post https://gitlab.*****.***/api/v4/runners: x509: certificate is valid for ingress.local, not gitlab.*****.***
PANIC: Failed to register the runner. You may be having network problems.
Registration attempt 15 of 30
Runtime platform                                    arch=amd64 os=linux pid=195 revision=8925d9a0 version=14.1.0
WARNING: Running in user-mode.
WARNING: The user-mode requires you to manually start builds processing:
WARNING: $ gitlab-runner run
WARNING: Use sudo for system-mode:
WARNING: $ sudo gitlab-runner...

What I already tried:
As in the Documentation explained I deployed the Certificate with a secret to the pod.

kubectl get secret -n gitlab-managed-apps:

NAME                                        TYPE                                  DATA   AGE
gitlab-domain-cert                          Opaque                                1      110m

I can verify that this works with kubectl exec

kubectl exec -n gitlab-managed-apps runner-gitlab-runner-5449984dfd-fmnhl -- cat /home/gitlab-runner/.gitlab-runner/certs/gitlab.*****.***.crt

the CRT File has the suggested Format

  (Your primary SSL certificate: your_domain_name.crt)
  (Your intermediate certificate)
  (Your root certificate)

I reviewed it many many many times to be sure it is the right certificate I also applied on my GitLab instance and also applied the Root certificate on my GitLab Instance. The same certificate works on a standalone in docker running GitLab Runner instance that is already registered and working.

To be honest I dont know how to check if there are maybe other Networking Issues within my Kubernetes Cluster. As I dont have root rights within the pod I cant even do a ping on my GitLab Instance.

I also tried to get my root cert registered as authority within the pod. There is a Blog entry that explains how you can achieve that, but it seems im to dumb to adapt the helmfile within the Management Project.


- name: gitlab

- name: runner
  namespace: gitlab-managed-apps
  chart: gitlab/gitlab-runner
  version: 0.31.0
  installed: true
  - name: ca-pemstore
    mountPath: /etc/ssl/certs/*****-ca.crt
    subPath: *****-ca.crt
    readOnly: false
    - name: ca-pemstore
        name: ca-pemstore
    - values.yaml.gotmpl

But all these adaptions wont make it through the CI/CD pipe. To be honest I just found this Blog today and gave it like 15 minutes of tryal and error, I should probably invest more time before giving up.