Ssh connection issue to gitlab-gitlab-shell

Installed Gitlab on k8s cluster with the provided helm chart.

Not using the provided ingress.

Service is behind a nginx proxy.

Following are the noticable settings in values.yaml:

global:
  common:
    labels:
      environment: poc

  edition: ee

  application:
    create: false
  
  hosts:
    domain: mydomain.tld 
    hostSuffix: gitlab
    https: true
    ssh: ~
    gitlab:
      name: gitlab.mydomain.tld

  ingress:
    enabled: false 
    configureCertmanager: false 
    provider: nginx
    tls: 
      enabled: true
      secretName: custom-domain-tld
    path: /
    pathType: Prefix

  appConfig:
    enableUsagePing: true
    enableSeatLink: true
    applicationSettingsCacheSeconds: 60
    defaultCanCreateGroup: true
    usernameChangingEnabled: true
    defaultProjectsFeatures:
      issues: true
      mergeRequests: true
      wiki: true
      snippets: true
      builds: true
    
  kas:
    enabled: true
    service:
      apiExternalPort: 8153

  spamcheck:
    enabled: true 

  shell:
    port: 8022
    tcp:
      proxyProtocol: false

certmanager:
  install: false 

nginx-ingress:
  enabled: false 
  tcpExternalConfig: "true"

Following the external nginx guide ( External NGINX Ingress Controller | GitLab ):
Configmap.yaml:

apiVersion: v1
kind: ConfigMap
metadata:
  name: tcp-gitlab-ingress
  namespace: ingress-nginx
data:
  8022: "gitlab/gitlab-gitlab-shell:8022"

is added, and in the deployment for the nginx following settings are provided as well:

apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/version: 1.2.0
  name: ingress-nginx-controller
  namespace: ingress-nginx
spec:
  ports:
  - appProtocol: http
    name: http
    port: 80
    protocol: TCP
    targetPort: http
  - appProtocol: https
    name: https
    port: 443
    protocol: TCP
    targetPort: https
  - name: ssh
    port: 8022
    protocol: TCP
    targetPort: 8022
  selector:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/name: ingress-nginx
  type: NodePort

and

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/version: 1.2.0
  name: ingress-nginx-controller
  namespace: ingress-nginx
spec:
  minReadySeconds: 0
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app.kubernetes.io/component: controller
      app.kubernetes.io/instance: ingress-nginx
      app.kubernetes.io/name: ingress-nginx
  template:
    metadata:
      labels:
        app.kubernetes.io/component: controller
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/name: ingress-nginx
    spec:
      containers:
      - args:
        - /nginx-ingress-controller
        - --election-id=ingress-controller-leader
        - --controller-class=k8s.io/ingress-nginx
        - --ingress-class=nginx
        - --configmap=$(POD_NAMESPACE)/ingress-nginx-controller
        - --validating-webhook=:8443
        - --validating-webhook-certificate=/usr/local/certificates/cert
        - --validating-webhook-key=/usr/local/certificates/key
        - --tcp-services-configmap=ingress-nginx/tcp-gitlab-ingress
        ports:
        - containerPort: 80
          name: http
          protocol: TCP
        - containerPort: 443
          name: https
          protocol: TCP
        - containerPort: 8443
          name: webhook
          protocol: TCP
        - containerPort: 8022
          name: ssh
          protocol: TCP

Resulting in the ingress service:

ingress-nginx-controller             NodePort    xx.xx.xxx.xxx   <none>        80:32026/TCP,443:3156/TCP,8022:30486/TCP

And the gitlab service:

gitlab-gitlab-shell                             ClusterIP   xx.xyz.xyz.xyz         8022/TCP
gitlab-webservice-default                       ClusterIP   xx.xx.xx.xx              8080/TCP,8181/TCP,8083/TCP

in the remote nginx (proxy) https and ssh trafic is proxied to the k8s cluster domain with the public ports for the workhorse and for the shell.

When attempting to ssh -Tvvv git@xx.xyz.xyz.xyz -p 8022 -i ~/.ssh/my_key from inside the cluster, this works.

When attempting to ssh using the (public) proxied domain, the traffic passes the nginx and is redirected tot the k8s domain, however this results in:

ssh -Tvvv git@my-public-domain.tld -p 8022 -i ~/.ssh/my_key
OpenSSH_8.6p1, LibreSSL 3.3.5
debug1: Reading configuration data /Users/it_is_I/.ssh/config
debug1: /Users/it_is_I/.ssh/config line 1: Applying options for *
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 21: include /etc/ssh/ssh_config.d/* matched no files
debug1: /etc/ssh/ssh_config line 54: Applying options for *
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/Users/it_is_I/.ssh/known_hosts'
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/Users/it_is_I/.ssh/known_hosts2'
debug1: Authenticator provider $SSH_SK_PROVIDER did not resolve; disabling
debug1: Connecting to my-public-domain.tld port 8022.
debug1: Connection established.
debug1: identity file /Users/it_is_I/.ssh/sven_gitlab type 0
debug1: identity file /Users/it_is_I/.ssh/sven_gitlab-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.6
kex_exchange_identification: Connection closed by remote host
Connection closed by zzz.zz.zz.zz port 8022

Using a worker-host ip or dns name, results in the same error.

As far as I understand the gitlab-shell can verify my key and if reached directly can handle the test/request.
When using the ingress settings in the k8s cluster, something go’s bonkers.

Please advice.
If needed I can provide extra logs and information.

Kind regards.