SSH keys no longer working, I think it was the 15.5 upgrade to gitlab-ce that broke it

I upgraded my gitlab-ce server today from the Ubuntu repo. My ssh keys are not working. Both of the keys that were offered in the attempt logged below are in my gitlab user’s Preferences. This was working yesterday, and the only thing that has been done to the gitlab server was the upgrade today. I did gitlab-ctl reconfigure and gitlab-ctl restart after I noticed the problem with no change.

elyograg@bilbo:/var/www/www.elyograg.org$ ssh -vT -o IdentitiesOnly=yes git@gitlab.elyograg.org 
OpenSSH_8.2p1 Ubuntu-4ubuntu0.5, OpenSSL 1.1.1f  31 Mar 2020
debug1: Reading configuration data /home/elyograg/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug1: Connecting to gitlab.elyograg.org [73.131.246.74] port 22.
debug1: Connection established.
debug1: identity file /home/elyograg/.ssh/id_rsa type 0
debug1: identity file /home/elyograg/.ssh/id_rsa-cert type -1
debug1: identity file /home/elyograg/.ssh/id_dsa type -1
debug1: identity file /home/elyograg/.ssh/id_dsa-cert type -1
debug1: identity file /home/elyograg/.ssh/id_ecdsa type 2
debug1: identity file /home/elyograg/.ssh/id_ecdsa-cert type -1
debug1: identity file /home/elyograg/.ssh/id_ecdsa_sk type -1
debug1: identity file /home/elyograg/.ssh/id_ecdsa_sk-cert type -1
debug1: identity file /home/elyograg/.ssh/id_ed25519 type -1
debug1: identity file /home/elyograg/.ssh/id_ed25519-cert type -1
debug1: identity file /home/elyograg/.ssh/id_ed25519_sk type -1
debug1: identity file /home/elyograg/.ssh/id_ed25519_sk-cert type -1
debug1: identity file /home/elyograg/.ssh/id_xmss type -1
debug1: identity file /home/elyograg/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.5
debug1: Remote protocol version 2.0, remote software version OpenSSH_9.0p1 Ubuntu-1ubuntu7
debug1: match: OpenSSH_9.0p1 Ubuntu-1ubuntu7 pat OpenSSH* compat 0x04000000
debug1: Authenticating to gitlab.elyograg.org:22 as 'git'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:nhFNfRLeSUlly7rnPKGTE5s5JACVJv2tleBE/T2DH8k
debug1: Host 'gitlab.elyograg.org' is known and matches the ECDSA host key.
debug1: Found key in /home/elyograg/.ssh/known_hosts:17
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey in after 134217728 blocks
debug1: Will attempt key: /home/elyograg/.ssh/id_rsa RSA SHA256:lCl/DGWbzat193bP3vFNorvSSwuGIu9deiyRMOQs4uQ
debug1: Will attempt key: /home/elyograg/.ssh/id_dsa 
debug1: Will attempt key: /home/elyograg/.ssh/id_ecdsa ECDSA SHA256:Oytu/NgUq6U63Cma0Xf7R+WInNz6iWHq9HiyIMH9cEQ
debug1: Will attempt key: /home/elyograg/.ssh/id_ecdsa_sk 
debug1: Will attempt key: /home/elyograg/.ssh/id_ed25519 
debug1: Will attempt key: /home/elyograg/.ssh/id_ed25519_sk 
debug1: Will attempt key: /home/elyograg/.ssh/id_xmss 
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,sk-ssh-ed25519@openssh.com,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ecdsa-sha2-nistp256@openssh.com,webauthn-sk-ecdsa-sha2-nistp256@openssh.com>
debug1: kex_input_ext_info: publickey-hostbound@openssh.com (unrecognised)
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,password
debug1: Next authentication method: publickey
debug1: Offering public key: /home/elyograg/.ssh/id_rsa RSA SHA256:lCl/DGWbzat193bP3vFNorvSSwuGIu9deiyRMOQs4uQ
debug1: Authentications that can continue: publickey,password
debug1: Trying private key: /home/elyograg/.ssh/id_dsa
debug1: Offering public key: /home/elyograg/.ssh/id_ecdsa ECDSA SHA256:Oytu/NgUq6U63Cma0Xf7R+WInNz6iWHq9HiyIMH9cEQ
debug1: Authentications that can continue: publickey,password
debug1: Trying private key: /home/elyograg/.ssh/id_ecdsa_sk
debug1: Trying private key: /home/elyograg/.ssh/id_ed25519
debug1: Trying private key: /home/elyograg/.ssh/id_ed25519_sk
debug1: Trying private key: /home/elyograg/.ssh/id_xmss
debug1: Next authentication method: password
git@gitlab.elyograg.org's password: 

The upgrade was from 15.4.3 to 15.5.0. In /var/log/dpkg.log:

2022-10-23 13:28:58 upgrade gitlab-ce:amd64 15.4.3-ce.0 15.5.0-ce.0
2022-10-23 13:28:58 status half-configured gitlab-ce:amd64 15.4.3-ce.0
2022-10-23 13:28:58 status unpacked gitlab-ce:amd64 15.4.3-ce.0
2022-10-23 13:28:58 status half-installed gitlab-ce:amd64 15.4.3-ce.0
2022-10-23 13:32:32 status unpacked gitlab-ce:amd64 15.5.0-ce.0
2022-10-23 13:32:45 configure gitlab-ce:amd64 15.5.0-ce.0 <none>
2022-10-23 13:32:45 status unpacked gitlab-ce:amd64 15.5.0-ce.0
2022-10-23 13:32:45 status half-configured gitlab-ce:amd64 15.5.0-ce.0
2022-10-23 13:36:44 status installed gitlab-ce:amd64 15.5.0-ce.0

Figured it out.

I had switched my server to have its IP address on br0 so I could set up bridged NICs in libvirt VMs.

This broke ucarp, which apparently cannot work on a bridged interface, and the gitlab name resolves to the VIP that two servers share with ucarp. So ssh connections were going to the other server, which does not have gitlab installed.

1 Like