Gitlab 12.10.1 ssh login, clone and pulls fail

I recently migrated to a new server using the backup/restore method. All seemed to be working well.

After pushing some code over the last few days today I tried a pull on another machine to update to the latest and got this error:

fatal: ‘/office/graphs.git’ does not appear to be a git repository
fatal: The remote end hung up unexpectedly

Hmm. Was working OK until a few weeks back.

Can we shell from the server to the new gitlab?

ssh -Tv -p 2224 git@gitlab.mydomain.com

Nope:

Permission denied (publickey).

ssh Back to the old gitlab (which is up to date)

No - permission denied again.

Can we shell to both old and new gitlab install as root?

ssh -p 2224 root@gitlab.mydomain.com

Yes - works perfectly. So it is JUST gitlab having a hissy fit.

Desktop work fine.

Have older RSA keys been deprecated? https://docs.gitlab.com/ee/ssh/README.html

Reset authorized_keys that do not seem to have been copied across with from the backup/restore (which is a bad bug IMHO)
As per the documentation:

gitlab-rake gitlab:shell:setup
gitlab-rake cache:clear
gitlab-rake gitlab:env:info
gitlab-rake gitlab:gitlab_shell:check
gitlab-rake gitlab:gitaly:check
gitlab-rake gitlab:sidekiq:check
gitlab-rake gitlab:app:check
gitlab-rake gitlab:check

All seems absolutely fine with no issues.

Can I connect? Nope.

Yes, it is an older version of ssh due to the server, but i connect perfectly as root - just won;t as git.

ssh -Tv -p 2224 git@gitlab.mydomain.com

OpenSSH_5.3p1, OpenSSL 1.0.1e-fips 11 Feb 2013
debug1: Reading configuration data /root/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
debug1: Connecting to gitlab.mydomain.com [1.2.3.4] port 2224.
debug1: Connection established.
debug1: permanently_set_uid: 0/0
debug1: identity file /root/.ssh/id_rsa type 1
debug1: identity file /root/.ssh/id_rsa-cert type -1
debug1: identity file /root/.ssh/id_dsa type -1
debug1: identity file /root/.ssh/id_dsa-cert type -1
debug1: identity file /root/.ssh/id_ecdsa type -1
debug1: identity file /root/.ssh/id_ecdsa-cert type -1
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.4p1 Debian-10+deb9u7
debug1: match: OpenSSH_7.4p1 Debian-10+deb9u7 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_5.3
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr hmac-sha1 none
debug1: kex: client->server aes128-ctr hmac-sha1 none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<2048<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug1: Host ‘[gitlab.mydomain.com]:2224’ is known and matches the RSA host key.
debug1: Found key in /root/.ssh/known_hosts:29
debug1: ssh_rsa_verify: signature correct
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Offering public key: /root/.ssh/id_rsa
debug1: Authentications that can continue: publickey
debug1: Trying private key: /root/.ssh/id_dsa
debug1: Trying private key: /root/.ssh/id_ecdsa
debug1: No more authentication methods to try.
Permission denied (publickey).

As root

ssh -v -p 2224 root@gitlab.mydomain.com
OpenSSH_5.3p1, OpenSSL 1.0.1e-fips 11 Feb 2013
debug1: Reading configuration data /root/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
debug1: Connecting to gitlab.mydomain.com [1.2.3.4] port 2224.
debug1: Connection established.
debug1: permanently_set_uid: 0/0
debug1: identity file /root/.ssh/id_rsa type 1
debug1: identity file /root/.ssh/id_rsa-cert type -1
debug1: identity file /root/.ssh/id_dsa type -1
debug1: identity file /root/.ssh/id_dsa-cert type -1
debug1: identity file /root/.ssh/id_ecdsa type -1
debug1: identity file /root/.ssh/id_ecdsa-cert type -1
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.4p1 Debian-10+deb9u7
debug1: match: OpenSSH_7.4p1 Debian-10+deb9u7 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_5.3
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr hmac-sha1 none
debug1: kex: client->server aes128-ctr hmac-sha1 none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<2048<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug1: Host ‘[gitlab.mydomain.com]:2224’ is known and matches the RSA host key.
debug1: Found key in /root/.ssh/known_hosts:29
debug1: ssh_rsa_verify: signature correct
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Offering public key: /root/.ssh/id_rsa
debug1: Server accepts key: pkalg ssh-rsa blen 277
debug1: read PEM private key done: type RSA
debug1: Authentication succeeded (publickey).
debug1: channel 0: new [client-session]
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.
debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0
debug1: Sending environment.
debug1: Sending env LANG = en_US.UTF-8
Linux gitlab 4.9.0-12-amd64 x86_64 GNU/Linux

Even worse - if I add a User to my ssh config and try git pull I still get

git pull
fatal: ‘/office/graphs.git’ does not appear to be a git repository
fatal: The remote end hung up unexpectedly

I have exactly the same issue on two other servers of the same type.

Cloning and pulling from https works fine.

Go figure. I can’t understand it, and there seem to be no errors in logs either.

I can only imagine that some ssh defaults have been updated internally in gitlab and I have not noticed.

Any help appreciated - I have updated code to pull and I can’t… !