Can't ssh to git@private-instance starting today

Yesterday I deployed a gitlab instance at gitlab.farzat.xyz. It was working perfectly yesterday, pushing using ssh. Today though ssh connections stopped working (I can connect to the server but can’t do git operations).

The output of ssh -vT git@gitlab.private-instance.com:

OpenSSH_8.2p1 Ubuntu-4ubuntu0.2, OpenSSL 1.1.1f  31 Mar 2020
debug1: Reading configuration data /home/farzat/.ssh/config
debug1: /home/farzat/.ssh/config line 1: Applying options for *
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug1: Connecting to gitlab.private-instance.com [2001:abcd:ef00:1234::1] port 22.
debug1: Connection established.
debug1: identity file /home/farzat/.ssh/id_rsa type 0
debug1: identity file /home/farzat/.ssh/id_rsa-cert type -1
debug1: identity file /home/farzat/.ssh/id_dsa type -1
debug1: identity file /home/farzat/.ssh/id_dsa-cert type -1
debug1: identity file /home/farzat/.ssh/id_ecdsa type -1
debug1: identity file /home/farzat/.ssh/id_ecdsa-cert type -1
debug1: identity file /home/farzat/.ssh/id_ecdsa_sk type -1
debug1: identity file /home/farzat/.ssh/id_ecdsa_sk-cert type -1
debug1: identity file /home/farzat/.ssh/id_ed25519 type -1
debug1: identity file /home/farzat/.ssh/id_ed25519-cert type -1
debug1: identity file /home/farzat/.ssh/id_ed25519_sk type -1
debug1: identity file /home/farzat/.ssh/id_ed25519_sk-cert type -1
debug1: identity file /home/farzat/.ssh/id_xmss type -1
debug1: identity file /home/farzat/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.2
debug1: Remote protocol version 2.0, remote software version OpenSSH_8.2p1 Ubuntu-4ubuntu0.2
debug1: match: OpenSSH_8.2p1 Ubuntu-4ubuntu0.2 pat OpenSSH* compat 0x04000000
debug1: Authenticating to gitlab.farzat.xyz:22 as 'git'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:5y1TNG1rS7UcD+OyWzGdwFws16OiOi/hy1XsP+iyapA
debug1: Host 'gitlab.private-instance.com' is known and matches the ECDSA host key.
debug1: Found key in /home/farzat/.ssh/known_hosts:69
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey in after 134217728 blocks
debug1: Will attempt key: /home/farzat/.ssh/id_rsa RSA SHA256:32nsXa4SCqvTkZGcF8WV+xePm5yr/VSiU7/SUOFVmLo agent
debug1: Will attempt key: farzat@farzat-legion5 RSA SHA256:GLOoIRdu2H/T3UvJPLyJiM0Myeq2ChD1JXuLAukMdaQ agent
debug1: Will attempt key: farzat@farzat-legion5 RSA SHA256:qGirjAXAcwKLGlTMSRLDPhlRFe1qvA0dmTwcMPn9ArU agent
debug1: Will attempt key: /home/farzat/.ssh/id_dsa
debug1: Will attempt key: /home/farzat/.ssh/id_ecdsa
debug1: Will attempt key: /home/farzat/.ssh/id_ecdsa_sk
debug1: Will attempt key: /home/farzat/.ssh/id_ed25519
debug1: Will attempt key: /home/farzat/.ssh/id_ed25519_sk
debug1: Will attempt key: /home/farzat/.ssh/id_xmss
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,sk-ssh-ed25519@openssh.com,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ecdsa-sha2-nistp256@openssh.com>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Offering public key: /home/farzat/.ssh/id_rsa RSA SHA256:32nsXa4SCqvTkZGcF8WV+xePm5yr/VSiU7/SUOFVmLo agent
debug1: Authentications that can continue: publickey
debug1: Offering public key: farzat@farzat-legion5 RSA SHA256:GLOoIRdu2H/T3UvJPLyJiM0Myeq2ChD1JXuLAukMdaQ agent
debug1: Authentications that can continue: publickey
debug1: Offering public key: farzat@farzat-legion5 RSA SHA256:qGirjAXAcwKLGlTMSRLDPhlRFe1qvA0dmTwcMPn9ArU agent
debug1: Authentications that can continue: publickey
debug1: Trying private key: /home/farzat/.ssh/id_dsa
debug1: Trying private key: /home/farzat/.ssh/id_ecdsa
debug1: Trying private key: /home/farzat/.ssh/id_ecdsa_sk
debug1: Trying private key: /home/farzat/.ssh/id_ed25519
debug1: Trying private key: /home/farzat/.ssh/id_ed25519_sk
debug1: Trying private key: /home/farzat/.ssh/id_xmss
debug1: No more authentication methods to try.
git@gitlab.private-instance.com: Permission denied (publickey).

Hi @Farzat07,

I’ve edited your post, since the FQDN / IPv6 and ports are public, avoiding potential abuse.

Regarding the SSH error, verify that the SSH key is put into a user profile on the GitLab server. Do that with a user account who has access to a project, not the admin account (just in case).

Also, to rule out other errors, create a Personal Access Token (PAT), and try cloning over https.

Cheers,
Michael

Thank you @dnsmichi for your help.

I figured out the problem - the git user on the server was locked and therefore the key was not accepted by the server. Simply adding a password to the git user on the server to unlock it fixed the issue.

Cheers,
Farzat

Hi,

I would recommend to disable the password login for the git user again after solving the SSH key problem. A common attack vector are SSH password logins. passwd -l should be able to do the trick, described in Disable a user's login without disabling the account - Unix & Linux Stack Exchange

Cheers,
Michael

Dear Michael,

Thanks for the advice.
I tried to do so, but the ssh key stopped working again. Once I ran passwd -u it worked again.
However I actually did disable ssh password logins globally on the server before even installing gitlab, so hopefully this should defend against the attacks. If you believe though that the security could be improved another way then I would be grateful for your help.

Cheers,
Farzat