Unable to gitlab connect over ssh

Summary

After new installation from source and after restoring data from backup i’m unable to access git over ssh (git clone over https works)

Steps to reproduce

git clone git@gitlab.alembiq.net:/larp/bistro.git

doesn’t matter if i try it from remote system, or from the server itself, results are same

What is the current bug behavior?

I’m unable to clone/pull/push over ssh

Relevant logs and/or screenshots

ssh -vT git@gitlab.alembiq.net
OpenSSH_7.9p1 Debian-10+deb10u2, OpenSSL 1.1.1d  10 Sep 2019
debug1: Reading configuration data /root/.ssh/config
debug1: /root/.ssh/config line 6: Applying options for gitlab.alembiq.net
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug1: Connecting to gitlab.alembiq.net [1..2.3.4] port 22.
debug1: Connection established.
debug1: identity file /root/.ssh/eir-2019-12-16 type 3
debug1: identity file /root/.ssh/eir-2019-12-16-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_7.9p1 Debian-10+deb10u2
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.9p1 Debian-10+deb10u2
debug1: match: OpenSSH_7.9p1 Debian-10+deb10u2 pat OpenSSH* compat 0x04000000
debug1: Authenticating to gitlab.alembiq.net:22 as 'git'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:1g1FzaQg123456fmj7T1Ga5Cui7sSgUhqZhpmb7OqxM
debug1: Host 'gitlab.alembiq.net' is known and matches the ECDSA host key.
debug1: Found key in /root/.ssh/known_hosts:1
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey after 134217728 blocks
debug1: Will attempt key: /root/.ssh/eir-2019-12-16 ED25519 SHA256:deB1ueu12345646hv1ROOaeIGsDx+CwxMSOPokQqaQo explicit
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Offering public key: /root/.ssh/eir-2019-12-16 ED25519 SHA256:deB1ueu12345646hv1ROOaeIGsDx+CwxMSOPokQqaQo explicit
debug1: Server accepts key: /root/.ssh/eir-2019-12-16 ED25519 SHA256:deB1ueu123456hv1ROOaeIGsDx+CwxMSOPokQqaQo explicit
debug1: Authentication succeeded (publickey).
Authenticated to gitlab.alembiq.net ([1.2.3.4]:22).
debug1: channel 0: new [client-session]
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.
debug1: pledge: network
debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0
debug1: Remote: /home/git/.ssh/authorized_keys:13: key options: command user-rc
debug1: Remote: /home/git/.ssh/authorized_keys:13: key options: command user-rc
debug1: Sending environment.
debug1: Sending env LC_MONETARY = cs_CZ.UTF-8
debug1: Sending env LANG = en_US.UTF-8
debug1: Sending env LC_MEASUREMENT = en_US.UTF-8
debug1: Sending env LC_TIME = en_US.UTF-8
debug1: Sending env LC_COLLATE = cs_CZ.UTF-8
debug1: Sending env LC_NUMERIC = cs_CZ.UTF-8
remote: 
remote: ========================================================================
remote: 
remote: Failed to get username: Internal API unreachable
remote: 
remote: ========================================================================
remote: 
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug1: client_input_channel_req: channel 0 rtype eow@openssh.com reply 0
debug1: channel 0: free: client-session, nchannels 1
Transferred: sent 2356, received 2784 bytes, in 0.1 seconds
Bytes per second: sent 23432.5, received 27689.3
debug1: Exit status 1

i can see my ssh key in /home/git/.ssh/authorized_keys

cat /home/git/.ssh/authorized_keys | grep AAAAC12345
command="/home/git/gitlab-shell/bin/gitlab-shell key-30",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty ssh-ed25519 AAAAC12345/67890

already tried to access rights there…

chown git.git /home/git -R
chmod 0711 /home/git
chmod 0700 /home/git/.ssh
chmod 0600 /home/git/.ssh/authorized_keys

Hi,

this sounds like a problem with gitlab-shell and the underlaying components talking to each other. Ensure that all services are running and operational. A possible hint could be in this issue but not a 100% sure about it.

Besides, I’d suggest to use the Omnibus packages instead of a source installation for better upgrades and troubleshooting.

Cheers,
Michael

this might be the solution, i will look into it after the weekend, when i’ll be back online… thank you; will let you know if it helped

BTW omnibus is not an option for me, as my server is VPS and i cannot modify some things, like the installer would like to (even when it has no reason to do so)

STDERR: sysctl: setting key "kernel.shmmax": Read-only file system                     
---- End output of sysctl -e -p /opt/gitlab/embedded/etc/90-omnibus-gitlab-kernel.shmmax.conf ----
Ran sysctl -e -p /opt/gitlab/embedded/etc/90-omnibus-gitlab-kernel.shmmax.conf returned 255           
sysctl: setting key "fs.protected_hardlinks": Read-only file system
sysctl: setting key "fs.protected_symlinks": Read-only file system

update: /home/git/gitlab/tmp/sockets/gitlab-workhorse.socket is just some old file, not active socket

2020/07/29 21:30:41 socat[8587] E connect(5, AF=1 "/home/git/gitlab/tmp/sockets/gitlab-workhorse.socket", 54): Connection refused

workhorse listens on tcp not socket, as it’s said in /etc/default/gitlab

gitlab_workhorse_options="-listenUmask 0 -listenNetwork tcp -listenAddr 127.0.0.1:8181 -authBackend http://127.0.0.1:8080"

so I guessed gitaly is trying to connect to the socket because things I’ve did based on above mentioned ticket https://gitlab.com/gitlab-org/gitaly/-/issues/2981
therefore i changed the url like so

url = "http://127.0.0.1:8181"

and lo! it works