Update to 15.11.3 breaks ssh key access

How to Use GitLab Self-managed
1

I have updated my self-hosted GitLab installation on Debian 11 to 15.11.3 today, and can now not push/pull or do anything else from any of the machines and accounts previously connected, all previously functioning as smooth as a baby kitten and unchanged in their configurations. All are now prompted for a password like so:

git@ourdomain.tld password:

Which shouldn’t be the case – there are ssh certificates, all without a password, that have been working perfectly fine until now. I double-checked permissions, and they seem all fine. Ran reconfigure, to no avail.

Anyone else having this problem, or has had it before, and can point me in the right direction?

2

What version were you on before the upgrade? I upgraded today from 15.11.2 to 15.11.3 and SSH access is working for me.

I would suggest:

gitlab-ctl reconfigure
gitlab-ctl restart
systemctl restart gitlab-runsvdir

and see if it helps. Could be an old process blocking it perhaps. The above commands are similar to what is suggested in the upgrade docs, when an upgraded instance isn’t behaving correctly after an upgrade and having some strange side-effects.

2 Likes
3

As you, from 15.11.2 – tried all three, all three run through without any issue, and the password request remains. There must be something I am not seeing…

4

Adding output of -vvv connection attempt here for reference:

ssh -Tvvv git@ourgitlab.url

debug1: Offering public key: /root/.ssh/id_ed25519 […]
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 51

Which is an unspecified authentication failure:

SSH_MSG_USERAUTH_REQUEST                50     [SSH-USERAUTH]
SSH_MSG_USERAUTH_FAILURE                51     [SSH-USERAUTH]
5

Created a new rsa key, uploaded that to Gitlab, and the result remains the same with the different key:

debug1: Next authentication method: publickey

debug1: Offering public key: /root/.ssh/id_rsa […]
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 51

Something is afoot.

6

Found it! The git user was locked, though not quite sure how that happened and have no time to find out. But! My solution:

nano /etc/shadow

Find the line with the git user on it, which likely looks like so:

git:!:19425::::::

or so

git:!!:19425::::::

One or two exclamation marks means that the account is locked. Replace the ‘!‘/’!!’ with something else such as a *, and schwuppdiwupp, git push/pull/clone and everything else works again as smoothly as before.

Your mileage may vary, of course, but to pinpoint your underlying cause, do this in two terminal windows:

On the machine you are trying to connect from:
ssh -Tvvv git@yourgitlab.url

On the machine you are trying to connect to:
tail -f /var/log/auth.log

1 Like