Can't access gitlab.com over SSH

theaussie86 · May 16, 2023, 5:40am

Hi everyone,
this might be a duplicate of Access via SSH stop working: Permission denied (publickey), but I decided to open this new ticket since I’m already using a Ed25519 based key and still can’ establish a connection over ssh.

Running ssh -Tvvv git@gitlab.com gives me the following output, which is not that helpful (at least to me) in solving my problem, so any help would be greatly appreciated:

cweissteiner@Christophs-MBP ~ % ssh -Tvvv git@gitlab.com

OpenSSH_9.0p1, LibreSSL 3.3.6

debug1: Reading configuration data /Users/cweissteiner/.ssh/config

debug1: /Users/cweissteiner/.ssh/config line 1: Applying options for *

debug1: Reading configuration data /etc/ssh/ssh_config

debug1: /etc/ssh/ssh_config line 21: include /etc/ssh/ssh_config.d/* matched no files

debug1: /etc/ssh/ssh_config line 54: Applying options for *

debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/Users/cweissteiner/.ssh/known_hosts'

debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/Users/cweissteiner/.ssh/known_hosts2'

debug1: Authenticator provider $SSH_SK_PROVIDER did not resolve; disabling

debug1: Connecting to gitlab.com port 22.

debug1: Connection established.

debug1: identity file /Users/cweissteiner/.ssh/id_rsa type 0

debug1: identity file /Users/cweissteiner/.ssh/id_rsa-cert type -1

debug1: Local version string SSH-2.0-OpenSSH_9.0

debug1: Remote protocol version 2.0, remote software version GitLab-SSHD

debug1: compat_banner: no match: GitLab-SSHD

debug3: fd 6 is O_NONBLOCK

debug1: Authenticating to gitlab.com:22 as 'git'

debug3: record_hostkey: found key type ED25519 in file /Users/cweissteiner/.ssh/known_hosts:1

debug3: record_hostkey: found key type RSA in file /Users/cweissteiner/.ssh/known_hosts:2

debug3: record_hostkey: found key type ECDSA in file /Users/cweissteiner/.ssh/known_hosts:3

debug3: load_hostkeys_file: loaded 3 keys from gitlab.com

debug1: load_hostkeys: fopen /Users/cweissteiner/.ssh/known_hosts2: No such file or directory

debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory

debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory

debug3: order_hostkeyalgs: have matching best-preference key type ssh-ed25519-cert-v01@openssh.com, using HostkeyAlgorithms verbatim

debug3: send packet: type 20

debug1: SSH2_MSG_KEXINIT sent

debug3: receive packet: type 20

debug1: SSH2_MSG_KEXINIT received

debug2: local client KEXINIT proposal

debug2: KEX algorithms: sntrup761x25519-sha512@openssh.com,curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,ext-info-c

debug2: host key algorithms: ssh-ed25519-cert-v01@openssh.com,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,rsa-sha2-512,rsa-sha2-256

debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com

debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com

debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1

debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1

debug2: compression ctos: none,zlib@openssh.com,zlib

debug2: compression stoc: none,zlib@openssh.com,zlib

debug2: languages ctos:

debug2: languages stoc:

debug2: first_kex_follows 0

debug2: reserved 0

debug2: peer server KEXINIT proposal

debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1

debug2: host key algorithms: ssh-dss,ecdsa-sha2-nistp256,ssh-ed25519,rsa-sha2-256,rsa-sha2-512,ssh-rsa

debug2: ciphers ctos: aes128-gcm@openssh.com,chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr

debug2: ciphers stoc: aes128-gcm@openssh.com,chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr

debug2: MACs ctos: hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1

debug2: MACs stoc: hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1

debug2: compression ctos: none

debug2: compression stoc: none

debug2: languages ctos:

debug2: languages stoc:

debug2: first_kex_follows 0

debug2: reserved 0

debug1: kex: algorithm: curve25519-sha256

debug1: kex: host key algorithm: ssh-ed25519

debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none

debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none

debug3: send packet: type 30

debug1: expecting SSH2_MSG_KEX_ECDH_REPLY

debug3: receive packet: type 31

debug1: SSH2_MSG_KEX_ECDH_REPLY received

debug1: Server host key: ssh-ed25519 SHA256:eUXGGm1YGsMAS7vkcx6JOJdOGHPem5gQp4taiCfCLB8

debug3: record_hostkey: found key type ED25519 in file /Users/cweissteiner/.ssh/known_hosts:1

debug3: record_hostkey: found key type RSA in file /Users/cweissteiner/.ssh/known_hosts:2

debug3: record_hostkey: found key type ECDSA in file /Users/cweissteiner/.ssh/known_hosts:3

debug3: load_hostkeys_file: loaded 3 keys from gitlab.com

debug1: load_hostkeys: fopen /Users/cweissteiner/.ssh/known_hosts2: No such file or directory

debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory

debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory

debug1: Host 'gitlab.com' is known and matches the ED25519 host key.

debug1: Found key in /Users/cweissteiner/.ssh/known_hosts:1

debug3: send packet: type 21

debug2: ssh_set_newkeys: mode 1

debug1: rekey out after 134217728 blocks

debug1: SSH2_MSG_NEWKEYS sent

debug1: expecting SSH2_MSG_NEWKEYS

debug3: receive packet: type 21

debug1: SSH2_MSG_NEWKEYS received

debug2: ssh_set_newkeys: mode 0

debug1: rekey in after 134217728 blocks

debug1: get_agent_identities: bound agent to hostkey

debug1: get_agent_identities: ssh_fetch_identitylist: agent contains no identities

debug1: Will attempt key: /Users/cweissteiner/.ssh/id_rsa RSA SHA256:X34UGXg1sgz8oMljb0JxlJBmkG9F4cFEFaSUaVmPZEY explicit

debug2: pubkey_prepare: done

debug3: send packet: type 5

debug3: receive packet: type 7

debug1: SSH2_MSG_EXT_INFO received

debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,sk-ssh-ed25519@openssh.com,sk-ecdsa-sha2-nistp256@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,rsa-sha2-256,rsa-sha2-512,ssh-rsa,ssh-dss>

debug3: receive packet: type 6

debug2: service_accept: ssh-userauth

debug1: SSH2_MSG_SERVICE_ACCEPT received

debug3: send packet: type 50

debug3: receive packet: type 51

debug1: Authentications that can continue: publickey

debug3: start over, passed a different list publickey

debug3: preferred publickey,keyboard-interactive,password

debug3: authmethod_lookup publickey

debug3: remaining preferred: keyboard-interactive,password

debug3: authmethod_is_enabled publickey

debug1: Next authentication method: publickey

debug1: Offering public key: /Users/cweissteiner/.ssh/id_rsa RSA SHA256:X34UGXg1sgz8oMljb0JxlJBmkG9F4cFEFaSUaVmPZEY explicit

debug3: send packet: type 50

debug2: we sent a publickey packet, wait for reply

debug3: receive packet: type 51

debug1: Authentications that can continue: publickey

debug2: we did not send a packet, disable method

debug1: No more authentication methods to try.

'git@gitlab.com': Permission denied (publickey).

paula.kokic · May 16, 2023, 7:19am

Hey there,

Do you maybe have more keys in your .ssh folder? To me looks like SSH is trying to log you in with your RSA key id_rsa, instead your id_ed25519 key (files by default have names according to the encryption algorithm).

Could you please:

check if the correct key (both public and private) are in your .ssh folder?
if both key pairs are in your .ssh folder, you probably need to add configuration that will tell your ssh client which key to use when connecting to which host. This you can do by creating a config file (no extension) with the following content:

Host gitlab.com
   HostName gitlab.com
   IdentityFile  ~/.ssh/id_ed25519
   User git

If you don’t need your other key pair (RSA), then you can also delete that one, and then it should work without the config file.

Also, stupid question, but - Did you add your public key to GitLab (via GitLab UI)? Pehraps it’s worth checking if it’s correct one.

Hope this helps!

nonformality · May 16, 2023, 3:47pm

Hey there @theaussie86 – in my case it helped to do this:

nano /etc/shadow

Find the line with the git user on it, and if it has one or two exclamation marks in it, like so:

git:!:19425::::::
git:!!:19425::::::

then replace the ‘!‘/’!!’ with something else such as a *, et voilà, git push/pull/clone and everything else works again as smoothly as before. Some more context and detail: Update to 15.11.3 breaks ssh key access

theaussie86 · May 18, 2023, 5:13am

Hi Paula,
this worked perfectly.
Thank you for your help.