Dear GitLab Community,
im fairly new to GItLab, working on it for a week now and I’m currently stuck with applying GitLab Runner within my BareMetal Kubernetes Cluster and I search your assistance to help me with this Issue.
tl;dr: GitLab runner cant register on my GitLab due invalid certificate authority. x509: certificate is valid for ingress.local as I use a Private Certificate
Kubernetes v1.18.3 - 2 Master, 3 Nodes
I use the GitLab Kubernetes Cluster Management Project, from it I deploy Ingress - Gitlab Runner - Prometehus and MetalLB. The CI/CD Pipe runs through and deploys everything within the suggested Namespace gitlab-managed-apps.
kubectl get all -n gitlab-managed-apps:
NAME READY STATUS RESTARTS AGE pod/ingress-nginx-ingress-controller-6cbf95f5d4-mvxdt 1/1 Running 0 3d20h pod/ingress-nginx-ingress-default-backend-77d64745d9-n75jj 1/1 Running 0 3d20h pod/metallb-controller-5c57bcd798-r6ctr 1/1 Running 0 3d18h pod/metallb-speaker-fmphd 1/1 Running 0 3d18h pod/metallb-speaker-fwqgc 1/1 Running 0 3d18h pod/metallb-speaker-fz4mm 1/1 Running 0 3d18h pod/metallb-speaker-p6xzp 1/1 Running 0 3d18h pod/metallb-speaker-z2zvz 1/1 Running 0 3d18h pod/prometheus-kube-state-metrics-7596c4bc64-hxk6n 1/1 Running 0 3d20h pod/prometheus-prometheus-server-597dc9d9c7-hq5r4 2/2 Running 0 3d20h pod/runner-gitlab-runner-5449984dfd-fmnhl 0/1 CrashLoopBackOff 16 103m NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/ingress-nginx-ingress-controller LoadBalancer 10.233.57.92 10.202.60.64 80:31695/TCP,443:30164/TCP 3d20h service/ingress-nginx-ingress-default-backend ClusterIP 10.233.35.32 <none> 80/TCP 3d20h service/prometheus-kube-state-metrics ClusterIP None <none> 80/TCP,81/TCP 3d20h service/prometheus-prometheus-server ClusterIP 10.233.36.218 <none> 80/TCP 3d20h NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE daemonset.apps/metallb-speaker 5 5 5 5 5 kubernetes.io/os=linux 3d18h NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/ingress-nginx-ingress-controller 1/1 1 1 3d20h deployment.apps/ingress-nginx-ingress-default-backend 1/1 1 1 3d20h deployment.apps/metallb-controller 1/1 1 1 3d18h deployment.apps/prometheus-kube-state-metrics 1/1 1 1 3d20h deployment.apps/prometheus-prometheus-server 1/1 1 1 3d20h deployment.apps/runner-gitlab-runner 0/1 1 0 3d20h
my GitLab Runner Pod crashes the whole time with following log output:
kubectl logs -n gitlab-managed-apps runner-gitlab-runner-5449984dfd-fmnhl
ERROR: Registering runner... failed runner=PJ-sm8tH status=couldn't execute POST against https://gitlab.*****.***/api/v4/runners: Post https://gitlab.*****.***/api/v4/runners: x509: certificate is valid for ingress.local, not gitlab.*****.*** PANIC: Failed to register the runner. You may be having network problems. Registration attempt 15 of 30 Runtime platform arch=amd64 os=linux pid=195 revision=8925d9a0 version=14.1.0 WARNING: Running in user-mode. WARNING: The user-mode requires you to manually start builds processing: WARNING: $ gitlab-runner run WARNING: Use sudo for system-mode: WARNING: $ sudo gitlab-runner...
What I already tried:
As in the Documentation explained I deployed the Certificate with a secret to the pod.
kubectl get secret -n gitlab-managed-apps: NAME TYPE DATA AGE gitlab-domain-cert Opaque 1 110m
I can verify that this works with kubectl exec
kubectl exec -n gitlab-managed-apps runner-gitlab-runner-5449984dfd-fmnhl -- cat /home/gitlab-runner/.gitlab-runner/certs/gitlab.*****.***.crt
the CRT File has the suggested Format
-----BEGIN CERTIFICATE----- (Your primary SSL certificate: your_domain_name.crt) -----END CERTIFICATE----- -----BEGIN CERTIFICATE----- (Your intermediate certificate) -----END CERTIFICATE----- -----BEGIN CERTIFICATE----- (Your root certificate) -----END CERTIFICATE-----
I reviewed it many many many times to be sure it is the right certificate I also applied on my GitLab instance and also applied the Root certificate on my GitLab Instance. The same certificate works on a standalone in docker running GitLab Runner instance that is already registered and working.
To be honest I dont know how to check if there are maybe other Networking Issues within my Kubernetes Cluster. As I dont have root rights within the pod I cant even do a ping on my GitLab Instance.
I also tried to get my root cert registered as authority within the pod. There is a Blog entry that explains how you can achieve that, but it seems im to dumb to adapt the helmfile within the Management Project.
repositories: - name: gitlab url: https://charts.gitlab.io releases: - name: runner namespace: gitlab-managed-apps chart: gitlab/gitlab-runner version: 0.31.0 installed: true volumeMounts: - name: ca-pemstore mountPath: /etc/ssl/certs/*****-ca.crt subPath: *****-ca.crt readOnly: false volumes: - name: ca-pemstore configMap: name: ca-pemstore values: - values.yaml.gotmpl
But all these adaptions wont make it through the CI/CD pipe. To be honest I just found this Blog today and gave it like 15 minutes of tryal and error, I should probably invest more time before giving up.