Need help: Gitlab Operator / on-prem Kubernetes Cluster (kubeadm) + Metallb / Problem Ingress: default backend - 404

Hi Gitlab-Fans,

I tried to install Gitlab via Operator on my kubeadm-cluster + metallb:

Before installing gitlab I already installed cert-manager and metrics like described in GitLab Operator | GitLab

My Gitlab Custom Ressource is pretty “basic”:

apiVersion: apps.gitlab.com/v1beta1
kind: GitLab
metadata:
  name: mygitlab
spec:
  chart:
    version: "5.7.0"
    values:
      global:
        hosts:
          domain: mygitlab.local # Provide a real base domain for GitLab. "gitlab." and "registry." will be exposed as subdomains.
        ingress:
          configureCertmanager: "true"
      certmanager-issuer:
          email: myrealmail@gmail.com

Ingress-controller got an load-balancer-ip from metallb:

k get svc -n gitlab-system:

NAME                                        TYPE           CLUSTER-IP       EXTERNAL-IP       PORT(S)                                   AGE
cm-acme-http-solver-gl7hl                   NodePort       10.98.17.48      <none>            8089:31719/TCP                            22h
cm-acme-http-solver-jw9jf                   NodePort       10.108.169.51    <none>            8089:31245/TCP                            22h
gitlab-controller-manager-metrics-service   ClusterIP      10.97.105.232    <none>            8443/TCP                                  24h
gitlab-webhook-service                      ClusterIP      10.109.232.25    <none>            443/TCP                                   24h
mygitlab-gitaly                             ClusterIP      None             <none>            8075/TCP                                  60m
mygitlab-gitlab-exporter                    ClusterIP      10.109.41.214    <none>            9168/TCP                                  60m
mygitlab-gitlab-shell                       ClusterIP      10.110.48.78     <none>            22/TCP                                    60m
mygitlab-minio                              ClusterIP      10.99.46.55      <none>            9000/TCP                                  60m
mygitlab-nginx-ingress-controller           LoadBalancer   10.106.213.152   192.168.200.243   80:30043/TCP,443:32522/TCP,22:31276/TCP   60m
mygitlab-nginx-ingress-controller-metrics   ClusterIP      10.109.109.119   <none>            10254/TCP                                 60m
mygitlab-nginx-ingress-defaultbackend       ClusterIP      10.108.135.3     <none>            80/TCP                                    60m
mygitlab-postgresql                         ClusterIP      10.98.214.81     <none>            5432/TCP                                  60m
mygitlab-postgresql-headless                ClusterIP      None             <none>            5432/TCP                                  60m
mygitlab-postgresql-metrics                 ClusterIP      10.96.185.9      <none>            9187/TCP                                  60m
mygitlab-redis-headless                     ClusterIP      None             <none>            6379/TCP                                  60m
mygitlab-redis-master                       ClusterIP      10.104.13.90     <none>            6379/TCP                                  60m
mygitlab-redis-metrics                      ClusterIP      10.96.157.138    <none>            9121/TCP                                  60m
mygitlab-registry                           ClusterIP      10.111.42.72     <none>            5000/TCP                                  60m
mygitlab-webservice-default                 ClusterIP      10.98.14.244     <none>            8080/TCP,8181/TCP                         60m

k get gitlabs -A

NAMESPACE       NAME       STATUS    VERSION
gitlab-system   mygitlab   Running   5.7.0

I also created the DNS-Records pointing to 192.168.200.243
gitlab.mygitlab.local → 192.168.200.243
registry.mygitlab.local → 192.168.200.243
minio.mygitlab.local → 192.168.200.243

Certs seem to be ok, but not sure:

k get issuer -A

NAMESPACE       NAME                       READY   AGE
gitlab-system   gitlab-selfsigned-issuer   True    26h
gitlab-system   mygitlab-issuer            True    5m8s

However if I connect with a browser to http://gitlab.mygitlab.local or https://gitlab.mygitlab.local I get default backend - 404

I also tried via curl within the cluster and it looks like this:

curl -IvL --insecure https://gitlab.mygitlab.local --resolve gitlab.mygitlab.local:443:192.168.200.243

* Added gitlab.mygitlab.local:443:192.168.200.243 to DNS cache
* Hostname gitlab.mygitlab.local was found in DNS cache
*   Trying 192.168.200.243:443...
* TCP_NODELAY set
* Connected to gitlab.mygitlab.local (192.168.200.243) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: O=Acme Co; CN=Kubernetes Ingress Controller Fake Certificate
*  start date: Feb  8 11:30:48 2022 GMT
*  expire date: Feb  8 11:30:48 2023 GMT
*  issuer: O=Acme Co; CN=Kubernetes Ingress Controller Fake Certificate
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x55672a81b880)
> HEAD / HTTP/2
> Host: gitlab.mygitlab.local
> user-agent: curl/7.68.0
> accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
< HTTP/2 404
HTTP/2 404
< date: Tue, 08 Feb 2022 12:37:45 GMT
date: Tue, 08 Feb 2022 12:37:45 GMT
< content-type: text/plain; charset=utf-8
content-type: text/plain; charset=utf-8
< content-length: 21
content-length: 21
< strict-transport-security: max-age=63072000
strict-transport-security: max-age=63072000
< referrer-policy: strict-origin-when-cross-origin
referrer-policy: strict-origin-when-cross-origin

<
* Connection #0 to host gitlab.mygitlab.local left intact

What am I doing wrong?
Thank you and Best Regards
Zen

@zenhighzer I am also facing same error, did you manage to resolve this issue? If yes, please let me know the solution.

Hi @srikantt,

please have a look here - there must be the solution:

BR
Zen

@zenhighzer and @mnielsen, Should I be applying TLS Option3 to overcome the trouble of “default backend - 404” and " * SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway."?

@srikantt

in my case the tls-option was not the problem. my problem was that i messed up the prefixes, so the ingress class, etc. were wrong.