Need help: Gitlab Operator / on-prem Kubernetes Cluster (kubeadm) + Metallb / Problem Ingress: default backend - 404

Hi Gitlab-Fans,

I tried to install Gitlab via Operator on my kubeadm-cluster + metallb:

Before installing gitlab I already installed cert-manager and metrics like described in GitLab Operator | GitLab

My Gitlab Custom Ressource is pretty “basic”:

kind: GitLab
  name: mygitlab
    version: "5.7.0"
          domain: mygitlab.local # Provide a real base domain for GitLab. "gitlab." and "registry." will be exposed as subdomains.
          configureCertmanager: "true"

Ingress-controller got an load-balancer-ip from metallb:

k get svc -n gitlab-system:

NAME                                        TYPE           CLUSTER-IP       EXTERNAL-IP       PORT(S)                                   AGE
cm-acme-http-solver-gl7hl                   NodePort      <none>            8089:31719/TCP                            22h
cm-acme-http-solver-jw9jf                   NodePort    <none>            8089:31245/TCP                            22h
gitlab-controller-manager-metrics-service   ClusterIP    <none>            8443/TCP                                  24h
gitlab-webhook-service                      ClusterIP    <none>            443/TCP                                   24h
mygitlab-gitaly                             ClusterIP      None             <none>            8075/TCP                                  60m
mygitlab-gitlab-exporter                    ClusterIP    <none>            9168/TCP                                  60m
mygitlab-gitlab-shell                       ClusterIP     <none>            22/TCP                                    60m
mygitlab-minio                              ClusterIP      <none>            9000/TCP                                  60m
mygitlab-nginx-ingress-controller           LoadBalancer   80:30043/TCP,443:32522/TCP,22:31276/TCP   60m
mygitlab-nginx-ingress-controller-metrics   ClusterIP   <none>            10254/TCP                                 60m
mygitlab-nginx-ingress-defaultbackend       ClusterIP     <none>            80/TCP                                    60m
mygitlab-postgresql                         ClusterIP     <none>            5432/TCP                                  60m
mygitlab-postgresql-headless                ClusterIP      None             <none>            5432/TCP                                  60m
mygitlab-postgresql-metrics                 ClusterIP      <none>            9187/TCP                                  60m
mygitlab-redis-headless                     ClusterIP      None             <none>            6379/TCP                                  60m
mygitlab-redis-master                       ClusterIP     <none>            6379/TCP                                  60m
mygitlab-redis-metrics                      ClusterIP    <none>            9121/TCP                                  60m
mygitlab-registry                           ClusterIP     <none>            5000/TCP                                  60m
mygitlab-webservice-default                 ClusterIP     <none>            8080/TCP,8181/TCP                         60m

k get gitlabs -A

gitlab-system   mygitlab   Running   5.7.0

I also created the DNS-Records pointing to
gitlab.mygitlab.local →
registry.mygitlab.local →
minio.mygitlab.local →

Certs seem to be ok, but not sure:

k get issuer -A

NAMESPACE       NAME                       READY   AGE
gitlab-system   gitlab-selfsigned-issuer   True    26h
gitlab-system   mygitlab-issuer            True    5m8s

However if I connect with a browser to http://gitlab.mygitlab.local or https://gitlab.mygitlab.local I get default backend - 404

I also tried via curl within the cluster and it looks like this:

curl -IvL --insecure https://gitlab.mygitlab.local --resolve gitlab.mygitlab.local:443:

* Added gitlab.mygitlab.local:443: to DNS cache
* Hostname gitlab.mygitlab.local was found in DNS cache
*   Trying
* Connected to gitlab.mygitlab.local ( port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: O=Acme Co; CN=Kubernetes Ingress Controller Fake Certificate
*  start date: Feb  8 11:30:48 2022 GMT
*  expire date: Feb  8 11:30:48 2023 GMT
*  issuer: O=Acme Co; CN=Kubernetes Ingress Controller Fake Certificate
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x55672a81b880)
> Host: gitlab.mygitlab.local
> user-agent: curl/7.68.0
> accept: */*
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
< HTTP/2 404
HTTP/2 404
< date: Tue, 08 Feb 2022 12:37:45 GMT
date: Tue, 08 Feb 2022 12:37:45 GMT
< content-type: text/plain; charset=utf-8
content-type: text/plain; charset=utf-8
< content-length: 21
content-length: 21
< strict-transport-security: max-age=63072000
strict-transport-security: max-age=63072000
< referrer-policy: strict-origin-when-cross-origin
referrer-policy: strict-origin-when-cross-origin

* Connection #0 to host gitlab.mygitlab.local left intact

What am I doing wrong?
Thank you and Best Regards

@zenhighzer I am also facing same error, did you manage to resolve this issue? If yes, please let me know the solution.

Hi @srikantt,

please have a look here - there must be the solution:


@zenhighzer and @mnielsen, Should I be applying TLS Option3 to overcome the trouble of “default backend - 404” and " * SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway."?


in my case the tls-option was not the problem. my problem was that i messed up the prefixes, so the ingress class, etc. were wrong.