Gitlab-sshd expected to run but nothing happens

How to Use GitLab Self-managed
1

On a self-managed instance (version 16.2.2, docker), I am trying to shift from OpenSSH to gitlab-sshd, followed the documentation from gitlab_sshd.html.

Here is my setup:

# /etc/gitlab/gitlab.rb
gitlab_sshd['enable'] = true
gitlab_sshd['listen_address'] = '[::]:2222' # also tried '0.0.0.0:2222'

# docker-compose.yml
...
  gitlab:
    ...
    ports:
      - 22:2222
    ...

After restarting the container, I can see the service is scheduled to start:

root@gitlab:/# ps auxf | grep ssh
root     10244  0.0  0.0   4016  2044 pts/0    S+   03:48   0:00  \_ grep --color=auto ssh
root        28  0.0  0.0   2652  1084 ?        Ss   03:29   0:00  \_ runsv sshd
root        29  0.0  0.0   2800  1060 ?        S    03:29   0:00  |   \_ svlogd -tt /var/log/gitlab/sshd
root        30  0.0  0.0  15432  8712 ?        S    03:29   0:00  |   \_ sshd: /usr/sbin/sshd -D -f /assets/sshd_config -e [listener] 0 of 100-200 startups
root       261  0.1  0.0   2652  1452 ?        Ss   03:30   0:01  \_ runsv gitlab-sshd
root       285  0.0  0.0   2800  1164 ?        S    03:30   0:00  |   \_ svlogd /var/log/gitlab/gitlab-sshd

but actually the executable gitlab-sshd is not running.

GitLab shell is installed, tried to run manually but nothing visible happens :

root@gitlab:/# /opt/gitlab/embedded/service/gitlab-shell/bin/gitlab-sshd --help
Usage of /opt/gitlab/embedded/service/gitlab-shell/bin/gitlab-sshd:
  -config-dir string
        The directory the config is in
root@gitlab:/# /opt/gitlab/embedded/service/gitlab-shell/bin/gitlab-sshd -config-dir /etc/gitlab/
FATA[0000] failed to load configuration from specified directory  error="open /etc/gitlab/config.yml: no such file or directory"
root@gitlab:/# /opt/gitlab/embedded/service/gitlab-shell/bin/gitlab-sshd -config-dir /opt/gitlab/embedded/service/gitlab-shell
root@gitlab:/# echo $?
1

Nothing listens on port 2222 (openssh server is still running by default):

root@gitlab:/# netstat -lauten | grep 22
tcp        0      0 127.0.0.1:9229          0.0.0.0:*               LISTEN      
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      
tcp        0      0 172.20.0.2:80           192.168.1.32:60722      TIME_WAIT   
tcp        0      0 127.0.0.1:9229          127.0.0.1:51690         ESTABLISHED 
tcp        0      0 127.0.0.1:51690         127.0.0.1:9229          ESTABLISHED 
tcp        0      0 :::22                   :::*                    LISTEN

The log file is empty:

root@gitlab:/# cat /var/log/gitlab/gitlab-sshd/current
root@gitlab:/# ls -la /var/log/gitlab/gitlab-sshd/
total 8
drwx------  2 git  root 4096 Aug 29 03:30 .
drwxr-xr-x 25 root root 4096 Aug 29 03:30 ..
lrwxrwxrwx  1 root root   37 Aug 29 03:30 config -> /opt/gitlab/sv/gitlab-sshd/log/config # no such file or directory
-rw-r--r--  1 root root    0 Aug 29 03:30 current
-rw-------  1 root root    0 Aug 29 03:30 lock

So… I don’t know where to go from there. Any clues?
Thanks!

2

While digging, I found this from gitlab.rb.template:

# gitlab-sshd outputs most logs to /var/log/gitlab/gitlab-shell/gitlab-shell.log.
# This directory only stores stdout/stderr output from the daemon.
# gitlab_sshd['log_directory'] = "/var/log/gitlab/gitlab-sshd/"

So I could find more logs in gitlab-shell.log, and the reason was permission issues on shared host
keys from the previous openssh server.

I hope this could help others solve or analyze their issues when switching to gitlab-sshd.