Installed Gitlab on k8s cluster with the provided helm chart.
Not using the provided ingress.
Service is behind a nginx proxy.
Following are the noticable settings in values.yaml:
global:
common:
labels:
environment: poc
edition: ee
application:
create: false
hosts:
domain: mydomain.tld
hostSuffix: gitlab
https: true
ssh: ~
gitlab:
name: gitlab.mydomain.tld
ingress:
enabled: false
configureCertmanager: false
provider: nginx
tls:
enabled: true
secretName: custom-domain-tld
path: /
pathType: Prefix
appConfig:
enableUsagePing: true
enableSeatLink: true
applicationSettingsCacheSeconds: 60
defaultCanCreateGroup: true
usernameChangingEnabled: true
defaultProjectsFeatures:
issues: true
mergeRequests: true
wiki: true
snippets: true
builds: true
kas:
enabled: true
service:
apiExternalPort: 8153
spamcheck:
enabled: true
shell:
port: 8022
tcp:
proxyProtocol: false
certmanager:
install: false
nginx-ingress:
enabled: false
tcpExternalConfig: "true"
Following the external nginx guide ( External NGINX Ingress Controller | GitLab ):
Configmap.yaml:
apiVersion: v1
kind: ConfigMap
metadata:
name: tcp-gitlab-ingress
namespace: ingress-nginx
data:
8022: "gitlab/gitlab-gitlab-shell:8022"
is added, and in the deployment for the nginx following settings are provided as well:
apiVersion: v1
kind: Service
metadata:
labels:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.2.0
name: ingress-nginx-controller
namespace: ingress-nginx
spec:
ports:
- appProtocol: http
name: http
port: 80
protocol: TCP
targetPort: http
- appProtocol: https
name: https
port: 443
protocol: TCP
targetPort: https
- name: ssh
port: 8022
protocol: TCP
targetPort: 8022
selector:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
type: NodePort
and
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.2.0
name: ingress-nginx-controller
namespace: ingress-nginx
spec:
minReadySeconds: 0
revisionHistoryLimit: 10
selector:
matchLabels:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
template:
metadata:
labels:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
spec:
containers:
- args:
- /nginx-ingress-controller
- --election-id=ingress-controller-leader
- --controller-class=k8s.io/ingress-nginx
- --ingress-class=nginx
- --configmap=$(POD_NAMESPACE)/ingress-nginx-controller
- --validating-webhook=:8443
- --validating-webhook-certificate=/usr/local/certificates/cert
- --validating-webhook-key=/usr/local/certificates/key
- --tcp-services-configmap=ingress-nginx/tcp-gitlab-ingress
ports:
- containerPort: 80
name: http
protocol: TCP
- containerPort: 443
name: https
protocol: TCP
- containerPort: 8443
name: webhook
protocol: TCP
- containerPort: 8022
name: ssh
protocol: TCP
Resulting in the ingress service:
ingress-nginx-controller NodePort xx.xx.xxx.xxx <none> 80:32026/TCP,443:3156/TCP,8022:30486/TCP
And the gitlab service:
gitlab-gitlab-shell ClusterIP xx.xyz.xyz.xyz 8022/TCP
gitlab-webservice-default ClusterIP xx.xx.xx.xx 8080/TCP,8181/TCP,8083/TCP
in the remote nginx (proxy) https and ssh trafic is proxied to the k8s cluster domain with the public ports for the workhorse and for the shell.
When attempting to ssh -Tvvv git@xx.xyz.xyz.xyz -p 8022 -i ~/.ssh/my_key from inside the cluster, this works.
When attempting to ssh using the (public) proxied domain, the traffic passes the nginx and is redirected tot the k8s domain, however this results in:
ssh -Tvvv git@my-public-domain.tld -p 8022 -i ~/.ssh/my_key
OpenSSH_8.6p1, LibreSSL 3.3.5
debug1: Reading configuration data /Users/it_is_I/.ssh/config
debug1: /Users/it_is_I/.ssh/config line 1: Applying options for *
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 21: include /etc/ssh/ssh_config.d/* matched no files
debug1: /etc/ssh/ssh_config line 54: Applying options for *
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/Users/it_is_I/.ssh/known_hosts'
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/Users/it_is_I/.ssh/known_hosts2'
debug1: Authenticator provider $SSH_SK_PROVIDER did not resolve; disabling
debug1: Connecting to my-public-domain.tld port 8022.
debug1: Connection established.
debug1: identity file /Users/it_is_I/.ssh/sven_gitlab type 0
debug1: identity file /Users/it_is_I/.ssh/sven_gitlab-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.6
kex_exchange_identification: Connection closed by remote host
Connection closed by zzz.zz.zz.zz port 8022
Using a worker-host ip or dns name, results in the same error.
As far as I understand the gitlab-shell can verify my key and if reached directly can handle the test/request.
When using the ingress settings in the k8s cluster, something go’s bonkers.
Please advice.
If needed I can provide extra logs and information.
Kind regards.