Gitlab agent for Kubernetes - x509: certificate signed by unknown authority

Infrastructure as Code & Cloud Native
1

Hi all,
I have a Omnibus Gitlab EE 15.10.7-ee.0 running with docker compose on premise
It works fine with Kubernetes Runner.

Issue
However, when I run kubectl get pods with GitLab CI using GitLab Agent, I get the following error. And the same error occurs when I add “–insecure-skip-tls-verify=true” option to kubectl.

$ kubectl config get-contexts
CURRENT   NAME                                     CLUSTER   AUTHINFO   NAMESPACE
          kimata/e-ink_weather_panel:pod-rollout   gitlab    agent:10   
$ kubectl config use-context kimata/e-ink_weather_panel:pod-rollout
$ kubectl get pods -A -v=10
I0514 05:45:46.148580      66 loader.go:373] Config loaded from file:  /builds/kimata/e-ink_weather_panel.tmp/KUBECONFIG
I0514 05:45:46.148814      66 merged_client_builder.go:163] Using in-cluster namespace
I0514 05:45:46.149281      66 round_trippers.go:466] curl -v -XGET  -H "Accept: application/json;g=apidiscovery.k8s.io;v=v2beta1;as=APIGroupDiscoveryList,application/json" -H "User-Agent: kubectl/v1.27.1 (linux/amd64) kubernetes/4c94112" -H "Authorization: Bearer <masked>" 'https://gitlab.green-rabbit.net/-/kubernetes-agent/k8s-proxy/api?timeout=32s'
I0514 05:45:46.154505      66 round_trippers.go:495] HTTP Trace: DNS Lookup for gitlab.green-rabbit.net resolved to [{192.168.0.27 }]
I0514 05:45:46.155070      66 round_trippers.go:510] HTTP Trace: Dial to tcp:192.168.0.27:443 succeed
I0514 05:45:46.209941      66 round_trippers.go:553] GET https://gitlab.green-rabbit.net/-/kubernetes-agent/k8s-proxy/api?timeout=32s  in 60 milliseconds
I0514 05:45:46.209967      66 round_trippers.go:570] HTTP Statistics: DNSLookup 4 ms Dial 0 ms TLSHandshake 54 ms Duration 60 ms
I0514 05:45:46.209976      66 round_trippers.go:577] Response Headers:
E0514 05:45:46.210052      66 memcache.go:265] couldn't get current server API group list: Get "https://gitlab.green-rabbit.net/-/kubernetes-agent/k8s-proxy/api?timeout=32s": tls: failed to verify certificate: x509: certificate signed by unknown authority
I0514 05:45:46.210062      66 cached_discovery.go:120] skipped caching discovery info due to Get "https://gitlab.green-rabbit.net/-/kubernetes-agent/k8s-proxy/api?timeout=32s": tls: failed to verify certificate: x509: certificate signed by unknown authority

I think the reason is that I am using EasyRSA signed keys in GitLab as follows, but I don’t know how to solve this problem.

    environment:                                                                                                                                                                                          
      GITLAB_OMNIBUS_CONFIG: |                                                                                                                                                                            
(snip)                                                                                                                                                                                                                                                                                                                                         
        nginx['enable'] = true                                                                                                                                                                            
        nginx['listen_port'] = 443                                                                                                                                                                        
        nginx['redirect_http_to_https'] = true                                                                                                                                                            
        nginx['ssl_certificate']     = "/etc/gitlab/ssl/XXX.crt"                                                                                                                      
        nginx['ssl_certificate_key']    = "/etc/gitlab/ssl/XX.key"

Measures taken
When I run Helm to install the Agent on a Kubernetes cluster, I use the --set-file config.caCert option, and the GitLab Agent does not output any errors.

% kubectl logs -l=app=gitlab-agent -n gitlab-agent-pod-rollout   
{"level":"info","time":"2023-05-14T04:26:08.268Z","msg":"Observability endpoint is up","mod_name":"observability","net_network":"tcp","net_address":"[::]:8080"}

After adding this option, gitlab-agent no longer outputs errors. However, kubectl in CI gives me an error.

I did something wrong ? Please give me some advice, thank you very much.

2

I found a workaround. The problem is that kasapp does not find the local ca.crt. Therefore, I need to create an image that contains it and use it to get it to work.

Specifically, build an image with the following Docker file and use it in docker-compose.

FROM gitlab/gitlab-ee:15.10.7-ee.0

COPY local-ca.crt /usr/local/share/ca-certificates/
RUN update-ca-certificates

I believe the correct solution is to add the equivalent of the --ca-cert-file option of agentk to kas.