I am currently trying to deploy gitlab CE via helm with an external ingress and cert-manager. I followed this guide and was already a bit confused at the beginning: The guide mentions secretName keys for webservice, registry, minio and kas. However, the official values.yaml file has predefined keys for the webservice, gitaly, praefect and the registry. Is it intentional that the values.yaml does not mention all possible keys which the helm chart understands?
However, let’s start with the webservice. This is the relevant setting in my values.yaml for the ingress:
## https://docs.gitlab.com/charts/charts/globals#configure-ingress-settings
ingress:
apiVersion: ""
configureCertmanager: false
provider: nginx
class: nginx
annotations:
cert-manager.io/cluster-issuer: letsencrypt-prod
kubernetes.io/ingress.class: nginx
kubernetes.io/tls-acme: true
enabled: true
tls:
enabled: false
secretName: RELEASE-gitlab-tls
path: /
pathType: Prefix
First, please note that i had to set the kubernetes.io/ingress-class annotation manually, since the setting of ingress.class did not have any effects on the resulting ingress object.
As a result, the ingress object is deployed to the cluster, but i do not have any certificates, certificaterequests or anything, and the pods refuse to startup, since the secrets containing the certificates do not exist.
All i see is this log excerpt from the cert-manager:
I0113 11:00:10.538572 1 sync.go:394] cert-manager/controller/ingress-shim "msg"="certificate resource is not owned by this object. refusing to update non-owned certificate resource for object" "related_resource_kind"="Certificate" "related_resource_name"="gitlab-tls" "related_resource_namespace"="gitlab" "related_resource_version"="v1" "resource_kind"="Ingress" "resource_name"="gitlab-registry" "resource_namespace"="gitlab" "resource_version"="v1"
I0113 11:00:10.538786 1 sync.go:394] cert-manager/controller/ingress-shim "msg"="certificate resource is not owned by this object. refusing to update non-owned certificate resource for object" "related_resource_kind"="Certificate" "related_resource_name"="gitlab-tls" "related_resource_namespace"="gitlab" "related_resource_version"="v1" "resource_kind"="Ingress" "resource_name"="gitlab-kas" "resource_namespace"="gitlab" "resource_version"="v1"
I0113 11:00:10.538842 1 sync.go:394] cert-manager/controller/ingress-shim "msg"="certificate resource is not owned by this object. refusing to update non-owned certificate resource for object" "related_resource_kind"="Certificate" "related_resource_name"="gitlab-tls" "related_resource_namespace"="gitlab" "related_resource_version"="v1" "resource_kind"="Ingress" "resource_name"="gitlab-webservice-default" "resource_namespace"="gitlab" "resource_version"="v1"
I0113 11:00:10.741269 1 trigger_controller.go:200] cert-manager/certificates-trigger "msg"="Certificate must be re-issued" "key"="gitlab/gitlab-tls" "message"="Issuing certificate as Secret was previously issued by ClusterIssuer.cert-manager.io/letsencrypt-staging" "reason"="IncorrectIssuer"
I0113 11:00:10.742783 1 conditions.go:203] Setting lastTransitionTime for Certificate "gitlab-tls" condition "Issuing" to 2023-01-13 11:00:10.741303559 +0000 UTC m=+2394.695905079
I0113 11:00:10.743207 1 conditions.go:192] Found status change for Certificate "gitlab-tls" condition "Ready": "True" -> "False"; setting lastTransitionTime to 2023-01-13 11:00:10.737134107 +0000 UTC m=+2394.691735637
I0113 11:00:10.797325 1 controller.go:162] cert-manager/certificates-trigger "msg"="re-queuing item due to optimistic locking on resource" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"gitlab-tls\": the object has been modified; please apply your changes to the latest version and try again" "key"="gitlab/gitlab-tls"
I0113 11:00:10.797579 1 trigger_controller.go:200] cert-manager/certificates-trigger "msg"="Certificate must be re-issued" "key"="gitlab/gitlab-tls" "message"="Issuing certificate as Secret was previously issued by ClusterIssuer.cert-manager.io/letsencrypt-staging" "reason"="IncorrectIssuer"
I0113 11:00:10.797596 1 conditions.go:203] Setting lastTransitionTime for Certificate "gitlab-tls" condition "Issuing" to 2023-01-13 11:00:10.797591396 +0000 UTC m=+2394.752192917
I0113 11:00:10.880430 1 controller.go:162] cert-manager/certificates-key-manager "msg"="re-queuing item due to optimistic locking on resource" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"gitlab-tls\": the object has been modified; please apply your changes to the latest version and try again" "key"="gitlab/gitlab-tls"
I0113 11:00:10.893818 1 conditions.go:263] Setting lastTransitionTime for CertificateRequest "gitlab-tls-xtn9q" condition "Approved" to 2023-01-13 11:00:10.893810838 +0000 UTC m=+2394.848412362
I0113 11:00:11.014696 1 conditions.go:263] Setting lastTransitionTime for CertificateRequest "gitlab-tls-xtn9q" condition "Ready" to 2023-01-13 11:00:11.01468762 +0000 UTC m=+2394.969289122
I0113 11:00:11.044753 1 conditions.go:263] Setting lastTransitionTime for CertificateRequest "gitlab-tls-xtn9q" condition "Ready" to 2023-01-13 11:00:11.044745834 +0000 UTC m=+2394.999347336
I0113 11:00:11.059525 1 controller.go:162] cert-manager/certificaterequests-issuer-acme "msg"="re-queuing item due to optimistic locking on resource" "error"="Operation cannot be fulfilled on certificaterequests.cert-manager.io \"gitlab-tls-xtn9q\": the object has been modified; please apply your changes to the latest version and try again" "key"="gitlab/gitlab-tls-xtn9q"
I0113 11:00:14.473600 1 acme.go:233] cert-manager/certificaterequests-issuer-acme/sign "msg"="certificate issued" "related_resource_kind"="Order" "related_resource_name"="gitlab-tls-xtn9q-201008078" "related_resource_namespace"="gitlab" "related_resource_version"="v1" "resource_kind"="CertificateRequest" "resource_name"="gitlab-tls-xtn9q" "resource_namespace"="gitlab" "resource_version"="v1"
I0113 11:00:14.473719 1 conditions.go:252] Found status change for CertificateRequest "gitlab-tls-xtn9q" condition "Ready": "False" -> "True"; setting lastTransitionTime to 2023-01-13 11:00:14.473713246 +0000 UTC m=+2398.428314748
I0113 11:00:14.554003 1 controller.go:162] cert-manager/certificates-readiness "msg"="re-queuing item due to optimistic locking on resource" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"gitlab-tls\": the object has been modified; please apply your changes to the latest version and try again" "key"="gitlab/gitlab-tls"
I0113 11:00:14.554469 1 conditions.go:192] Found status change for Certificate "gitlab-tls" condition "Ready": "False" -> "True"; setting lastTransitionTime to 2023-01-13 11:00:14.554460404 +0000 UTC m=+2398.509061928
I0113 11:00:14.584253 1 controller.go:162] cert-manager/certificates-key-manager "msg"="re-queuing item due to optimistic locking on resource" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"gitlab-tls\": the object has been modified; please apply your changes to the latest version and try again" "key"="gitlab/gitlab-tls"
I0113 12:36:38.173778 1 util.go:84] cert-manager/controller/certificaterequests-issuer-acme/handleOwnedResource "msg"="owning resource not found in cache" "related_resource_kind"="CertificateRequest" "related_resource_name"="gitlab-tls-xtn9q" "related_resource_namespace"="gitlab" "resource_kind"="Order" "resource_name"="gitlab-tls-xtn9q-201008078" "resource_namespace"="gitlab" "resource_version"="v1"
E0113 12:36:38.173880 1 controller.go:176] cert-manager/orders "msg"="order in work queue no longer exists" "error"="order.acme.cert-manager.io \"gitlab-tls-xtn9q-201008078\" not found"
I0113 12:36:38.176363 1 util.go:84] cert-manager/controller/certificaterequests-issuer-acme/handleOwnedResource "msg"="owning resource not found in cache" "related_resource_kind"="CertificateRequest" "related_resource_name"="gitlab-tls-z76w7" "related_resource_namespace"="gitlab" "resource_kind"="Order" "resource_name"="gitlab-tls-z76w7-201008078" "resource_namespace"="gitlab" "resource_version"="v1"
E0113 12:36:38.176400 1 controller.go:176] cert-manager/orders "msg"="order in work queue no longer exists" "error"="order.acme.cert-manager.io \"gitlab-tls-z76w7-201008078\" not found"
E0113 12:36:38.176413 1 controller.go:176] cert-manager/orders "msg"="order in work queue no longer exists" "error"="order.acme.cert-manager.io \"gitlab-tls-slpr5-3018188670\" not found"
I0113 12:36:38.176435 1 util.go:84] cert-manager/controller/certificaterequests-issuer-acme/handleOwnedResource "msg"="owning resource not found in cache" "related_resource_kind"="CertificateRequest" "related_resource_name"="gitlab-tls-slpr5" "related_resource_namespace"="gitlab" "resource_kind"="Order" "resource_name"="gitlab-tls-slpr5-3018188670" "resource_namespace"="gitlab" "resource_version"="v1"
I0113 12:36:38.177022 1 util.go:84] cert-manager/controller/certificaterequests-issuer-acme/handleOwnedResource "msg"="owning resource not found in cache" "related_resource_kind"="CertificateRequest" "related_resource_name"="gitlab-tls-kmw9k" "related_resource_namespace"="gitlab" "resource_kind"="Order" "resource_name"="gitlab-tls-kmw9k-3412373284" "resource_namespace"="gitlab" "resource_version"="v1"
E0113 12:36:38.177040 1 controller.go:176] cert-manager/orders "msg"="order in work queue no longer exists" "error"="order.acme.cert-manager.io \"gitlab-tls-kmw9k-3412373284\" not found"
It seems that the Certificate-manager gets a certificate request from gitlab, however, there are no certificaterequests in the gitlab namespace:
❯ k ns gitlab
Active namespace is "gitlab".
❯ k get certificaterequest
No resources found in gitlab namespace.
❯ k ns ingress-nginx
Active namespace is "ingress-nginx".
❯ k get certificaterequest
No resources found in ingress-nginx namespace.
According to the certmanager log, there have been some certificate orders, but are not present anymore. I am a bit unsure how to proceed from here, since i do not have any idea why the certmanager or gitlab would delete any already existing orders. Also, because the documentation does not really match with the actual situation: What is the recommended way to proceed from here?