Dear GitLab Community,
im fairly new to GItLab, working on it for a week now and I’m currently stuck with applying GitLab Runner within my BareMetal Kubernetes Cluster and I search your assistance to help me with this Issue.
tl;dr: GitLab runner cant register on my GitLab due invalid certificate authority. x509: certificate is valid for ingress.local as I use a Private Certificate
My Environment:
Kubernetes v1.18.3 - 2 Master, 3 Nodes
Helm v3.3.4
GitLab 14.3.2-ee
My Issue:
I use the GitLab Kubernetes Cluster Management Project, from it I deploy Ingress - Gitlab Runner - Prometehus and MetalLB. The CI/CD Pipe runs through and deploys everything within the suggested Namespace gitlab-managed-apps.
kubectl get all -n gitlab-managed-apps:
NAME READY STATUS RESTARTS AGE
pod/ingress-nginx-ingress-controller-6cbf95f5d4-mvxdt 1/1 Running 0 3d20h
pod/ingress-nginx-ingress-default-backend-77d64745d9-n75jj 1/1 Running 0 3d20h
pod/metallb-controller-5c57bcd798-r6ctr 1/1 Running 0 3d18h
pod/metallb-speaker-fmphd 1/1 Running 0 3d18h
pod/metallb-speaker-fwqgc 1/1 Running 0 3d18h
pod/metallb-speaker-fz4mm 1/1 Running 0 3d18h
pod/metallb-speaker-p6xzp 1/1 Running 0 3d18h
pod/metallb-speaker-z2zvz 1/1 Running 0 3d18h
pod/prometheus-kube-state-metrics-7596c4bc64-hxk6n 1/1 Running 0 3d20h
pod/prometheus-prometheus-server-597dc9d9c7-hq5r4 2/2 Running 0 3d20h
pod/runner-gitlab-runner-5449984dfd-fmnhl 0/1 CrashLoopBackOff 16 103m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/ingress-nginx-ingress-controller LoadBalancer 10.233.57.92 10.202.60.64 80:31695/TCP,443:30164/TCP 3d20h
service/ingress-nginx-ingress-default-backend ClusterIP 10.233.35.32 <none> 80/TCP 3d20h
service/prometheus-kube-state-metrics ClusterIP None <none> 80/TCP,81/TCP 3d20h
service/prometheus-prometheus-server ClusterIP 10.233.36.218 <none> 80/TCP 3d20h
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
daemonset.apps/metallb-speaker 5 5 5 5 5 kubernetes.io/os=linux 3d18h
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/ingress-nginx-ingress-controller 1/1 1 1 3d20h
deployment.apps/ingress-nginx-ingress-default-backend 1/1 1 1 3d20h
deployment.apps/metallb-controller 1/1 1 1 3d18h
deployment.apps/prometheus-kube-state-metrics 1/1 1 1 3d20h
deployment.apps/prometheus-prometheus-server 1/1 1 1 3d20h
deployment.apps/runner-gitlab-runner 0/1 1 0 3d20h
my GitLab Runner Pod crashes the whole time with following log output:
kubectl logs -n gitlab-managed-apps runner-gitlab-runner-5449984dfd-fmnhl
ERROR: Registering runner... failed runner=PJ-sm8tH status=couldn't execute POST against https://gitlab.*****.***/api/v4/runners: Post https://gitlab.*****.***/api/v4/runners: x509: certificate is valid for ingress.local, not gitlab.*****.***
PANIC: Failed to register the runner. You may be having network problems.
Registration attempt 15 of 30
Runtime platform arch=amd64 os=linux pid=195 revision=8925d9a0 version=14.1.0
WARNING: Running in user-mode.
WARNING: The user-mode requires you to manually start builds processing:
WARNING: $ gitlab-runner run
WARNING: Use sudo for system-mode:
WARNING: $ sudo gitlab-runner...
What I already tried:
As in the Documentation explained I deployed the Certificate with a secret to the pod.
kubectl get secret -n gitlab-managed-apps:
NAME TYPE DATA AGE
gitlab-domain-cert Opaque 1 110m
I can verify that this works with kubectl exec
kubectl exec -n gitlab-managed-apps runner-gitlab-runner-5449984dfd-fmnhl -- cat /home/gitlab-runner/.gitlab-runner/certs/gitlab.*****.***.crt
the CRT File has the suggested Format
-----BEGIN CERTIFICATE-----
(Your primary SSL certificate: your_domain_name.crt)
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
(Your intermediate certificate)
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
(Your root certificate)
-----END CERTIFICATE-----
I reviewed it many many many times to be sure it is the right certificate I also applied on my GitLab instance and also applied the Root certificate on my GitLab Instance. The same certificate works on a standalone in docker running GitLab Runner instance that is already registered and working.
To be honest I dont know how to check if there are maybe other Networking Issues within my Kubernetes Cluster. As I dont have root rights within the pod I cant even do a ping on my GitLab Instance.
I also tried to get my root cert registered as authority within the pod. There is a Blog entry that explains how you can achieve that, but it seems im to dumb to adapt the helmfile within the Management Project.
applications/gitlab-runner/helmfile.yaml
repositories:
- name: gitlab
url: https://charts.gitlab.io
releases:
- name: runner
namespace: gitlab-managed-apps
chart: gitlab/gitlab-runner
version: 0.31.0
installed: true
volumeMounts:
- name: ca-pemstore
mountPath: /etc/ssl/certs/*****-ca.crt
subPath: *****-ca.crt
readOnly: false
volumes:
- name: ca-pemstore
configMap:
name: ca-pemstore
values:
- values.yaml.gotmpl
But all these adaptions wont make it through the CI/CD pipe. To be honest I just found this Blog today and gave it like 15 minutes of tryal and error, I should probably invest more time before giving up.