Ingress does not have an active endpoint

Ingress was constantly hanging in the kubernetes management page, so I decided to check the logs:

$ kubectl logs -n gitlab-managed-apps ingress-nginx-ingress-controller-69db88fd-djwvk
-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:       0.25.1
  Build:         git-5179893a9
  Repository:    https://github.com/kubernetes/ingress-nginx/
  nginx version: openresty/1.15.8.1

-------------------------------------------------------------------------------

I1224 13:25:48.832978       6 flags.go:192] Watching for Ingress class: nginx
W1224 13:25:48.833492       6 flags.go:221] SSL certificate chain completion is disabled (--enable-ssl-chain-completion=false)
nginx version: openresty/1.15.8.1
W1224 13:25:48.917348       6 client_config.go:541] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I1224 13:25:48.917696       6 main.go:183] Creating API client for https://10.152.183.1:443
I1224 13:25:49.503235       6 main.go:227] Running in Kubernetes cluster version v1.17 (v1.17.0) - git (clean) commit 70132b0f130acc0bed193d9ba59dd186f0e634cf - platform linux/amd64
I1224 13:25:49.534092       6 main.go:91] Validated gitlab-managed-apps/ingress-nginx-ingress-default-backend as the default backend.
I1224 13:25:55.591425       6 main.go:102] Created fake certificate with PemFileName: /etc/ingress-controller/ssl/default-fake-certificate.pem
W1224 13:25:56.821721       6 store.go:616] Unexpected error reading configuration configmap: configmaps "ingress-nginx-ingress-controller" not found
I1224 13:25:57.169398       6 nginx.go:274] Starting NGINX Ingress controller
I1224 13:25:58.486354       6 event.go:258] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"staging-test-ci-8qx4vt", Name:"test-ci", UID:"16f0d1d8-0995-410c-b9c9-19dbf690103f", APIVersion:"networking.k8s.io/v1beta1", ResourceVersion:"29058782", FieldPath:""}): type: 'Normal' reason: 'CREATE' Ingress staging-test-ci-8qx4vt/test-ci
I1224 13:25:58.486784       6 event.go:258] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"service-feat-dock-uupba1", Name:"service-feat-dock-uupba1", UID:"d751c2ab-6c18-11e9-b34f-fef9812c353d", APIVersion:"networking.k8s.io/v1beta1", ResourceVersion:"29058770", FieldPath:""}): type: 'Normal' reason: 'CREATE' Ingress service-feat-dock-uupba1/service-feat-dock-uupba1
I1224 13:25:58.487123       6 event.go:258] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"staging-develop-7qd8v5", Name:"staging-develop-7qd8v5", UID:"8473909e-11fa-4563-b7cc-4ad0f5f96c9f", APIVersion:"networking.k8s.io/v1beta1", ResourceVersion:"29058776", FieldPath:""}): type: 'Normal' reason: 'CREATE' Ingress staging-develop-7qd8v5/staging-develop-7qd8v5
I1224 13:25:58.487539       6 event.go:258] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"staging-feat-cont-ov72jj", Name:"staging-feat-cont-ov72jj", UID:"e37e06e1-02a8-46d0-9d45-c9b9206c255c", APIVersion:"networking.k8s.io/v1beta1", ResourceVersion:"29058774", FieldPath:""}): type: 'Normal' reason: 'CREATE' Ingress staging-feat-cont-ov72jj/staging-feat-cont-ov72jj
I1224 13:25:58.488000       6 event.go:258] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"staging-feat-dock-0p4rjk", Name:"staging-feat-dock-0p4rjk", UID:"6b78440d-6a7f-11e9-b34f-fef9812c353d", APIVersion:"networking.k8s.io/v1beta1", ResourceVersion:"29058772", FieldPath:""}): type: 'Normal' reason: 'CREATE' Ingress staging-feat-dock-0p4rjk/staging-feat-dock-0p4rjk
I1224 13:25:58.488412       6 event.go:258] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"staging-feat-queu-ia6adv", Name:"staging-feat-queu-ia6adv", UID:"05b13cf5-f78e-42a0-a4c2-307ea25795bc", APIVersion:"networking.k8s.io/v1beta1", ResourceVersion:"29058771", FieldPath:""}): type: 'Normal' reason: 'CREATE' Ingress staging-feat-queu-ia6adv/staging-feat-queu-ia6adv
I1224 13:25:58.488556       6 event.go:258] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"staging-feat-webr-azedd9", Name:"feat-webrtc-client", UID:"34e327e5-24c1-4144-b369-cb7ff56cae8b", APIVersion:"networking.k8s.io/v1beta1", ResourceVersion:"29058779", FieldPath:""}): type: 'Normal' reason: 'CREATE' Ingress staging-feat-webr-azedd9/feat-webrtc-client
I1224 13:25:58.501730       6 nginx.go:318] Starting NGINX process
I1224 13:25:58.502067       6 leaderelection.go:235] attempting to acquire leader lease  gitlab-managed-apps/ingress-controller-leader-nginx...
W1224 13:25:58.567811       6 controller.go:384] Service "gitlab-managed-apps/ingress-nginx-ingress-default-backend" does not have any active Endpoint
W1224 13:25:58.596937       6 controller.go:878] Service "service-feat-dock-uupba1/uwsgi-service" does not have any active Endpoint.
W1224 13:25:58.596985       6 controller.go:878] Service "staging-develop-7qd8v5/uwsgi-service" does not have any active Endpoint.
W1224 13:25:58.597004       6 controller.go:878] Service "staging-feat-cont-ov72jj/uwsgi-service" does not have any active Endpoint.
W1224 13:25:58.597021       6 controller.go:878] Service "staging-feat-webr-azedd9/uwsgi-service" does not have any active Endpoint.
I1224 13:25:58.597157       6 controller.go:133] Configuration changes detected, backend reload required.
I1224 13:25:59.876919       6 leaderelection.go:245] successfully acquired lease gitlab-managed-apps/ingress-controller-leader-nginx
I1224 13:25:59.877350       6 status.go:86] new leader elected: ingress-nginx-ingress-controller-69db88fd-djwvk
E1224 13:26:01.799403       6 checker.go:41] healthcheck error: Get http+unix://nginx-status/healthz: dial unix /tmp/nginx-status-server.sock: connect: no such file or directory
E1224 13:26:02.414417       6 checker.go:41] healthcheck error: Get http+unix://nginx-status/healthz: dial unix /tmp/nginx-status-server.sock: connect: no such file or directory
I1224 13:26:03.946184       6 controller.go:149] Backend successfully reloaded.
I1224 13:26:03.946285       6 controller.go:158] Initial sync, sleeping for 1 second.
W1224 13:26:04.946899       6 controller.go:176] Dynamic reconfiguration failed: Post http+unix://nginx-status/configuration/backends: dial unix /tmp/nginx-status-server.sock: connect: no such file or directory
E1224 13:26:04.946927       6 controller.go:180] Unexpected failure reconfiguring NGINX:
Post http+unix://nginx-status/configuration/backends: dial unix /tmp/nginx-status-server.sock: connect: no such file or directory
W1224 13:26:04.946938       6 queue.go:130] requeuing initial-sync, err Post http+unix://nginx-status/configuration/backends: dial unix /tmp/nginx-status-server.sock: connect: no such file or directory
W1224 13:26:04.946988       6 controller.go:384] Service "gitlab-managed-apps/ingress-nginx-ingress-default-backend" does not have any active Endpoint
W1224 13:26:04.947056       6 controller.go:878] Service "staging-feat-webr-azedd9/uwsgi-service" does not have any active Endpoint.
I1224 13:26:04.947141       6 controller.go:133] Configuration changes detected, backend reload required.
I1224 13:26:05.024986       6 controller.go:149] Backend successfully reloaded.
I1224 13:26:05.025023       6 controller.go:158] Initial sync, sleeping for 1 second.
W1224 13:26:06.025640       6 controller.go:176] Dynamic reconfiguration failed: Post http+unix://nginx-status/configuration/backends: dial unix /tmp/nginx-status-server.sock: connect: no such file or directory
E1224 13:26:06.025677       6 controller.go:180] Unexpected failure reconfiguring NGINX:
Post http+unix://nginx-status/configuration/backends: dial unix /tmp/nginx-status-server.sock: connect: no such file or directory
W1224 13:26:06.025687       6 queue.go:130] requeuing staging-test-ci-8qx4vt/realtime-service, err Post http+unix://nginx-status/configuration/backends: dial unix /tmp/nginx-status-server.sock: connect: no such file or directory
W1224 13:26:08.280856       6 controller.go:384] Service "gitlab-managed-apps/ingress-nginx-ingress-default-backend" does not have any active Endpoint
W1224 13:26:08.281048       6 controller.go:878] Service "staging-feat-webr-azedd9/uwsgi-service" does not have any active Endpoint.
I1224 13:26:08.281230       6 controller.go:133] Configuration changes detected, backend reload required.
I1224 13:26:08.355022       6 controller.go:149] Backend successfully reloaded.
I1224 13:26:08.355058       6 controller.go:158] Initial sync, sleeping for 1 second.
W1224 13:26:11.613806       6 controller.go:384] Service "gitlab-managed-apps/ingress-nginx-ingress-default-backend" does not have any active Endpoint
1 Like