OpenSsh 8.5p1-1 potential problem ? SSH keys on Gitlab

Hello as of yesterday every time I try to interact gitlab - using my regular ssh keys I get the following warning.

client_global_hostkeys_private_confirm: server gave bad signature for RSA key 0

  • I am a MacOSX user (Big Sur)
  • OpenSSH_8.5p1-1, LibreSSL 2.7.3

when I try to git pull or push a change to my repos - I always get the above error.

Initially I had 2 ssh keys one for day job and another for personal projects (on gitlab CE). I thought it might be that. I tried only adding to my SSH agent - the main one (id_rsa) but I still get the warnings.

It seems other users on other operating systems have exactly this issue

See here:https://www.reddit.com/r/archlinux/comments/lyazre/openssh_update_causes_problems/

Any tips?

Hi,

Just out of interest how long has it been since your ssh-rsa key was generated?

I would expect it has something to do with this: https://www.openssh.com/txt/release-8.5

Future deprecation notice

It is now possible[1] to perform chosen-prefix attacks against the
SHA-1 algorithm for less than USD$50K.

In the SSH protocol, the “ssh-rsa” signature scheme uses the SHA-1
hash algorithm in conjunction with the RSA public key algorithm.
OpenSSH will disable this signature scheme by default in the near
future.

Note that the deactivation of “ssh-rsa” signatures does not necessarily
require cessation of use for RSA keys. In the SSH protocol, keys may be
capable of signing using multiple algorithms. In particular, “ssh-rsa”
keys are capable of signing using “rsa-sha2-256” (RSA/SHA256),
“rsa-sha2-512” (RSA/SHA512) and “ssh-rsa” (RSA/SHA1). Only the last of
these is being turned off by default.

This algorithm is unfortunately still used widely despite the
existence of better alternatives, being the only remaining public key
signature algorithm specified by the original SSH RFCs that is still
enabled by default.

The better alternatives include:

  • The RFC8332 RSA SHA-2 signature algorithms rsa-sha2-256/512. These
    algorithms have the advantage of using the same key type as
    “ssh-rsa” but use the safe SHA-2 hash algorithms. These have been
    supported since OpenSSH 7.2 and are already used by default if the
    client and server support them.

  • The RFC8709 ssh-ed25519 signature algorithm. It has been supported
    in OpenSSH since release 6.5.

  • The RFC5656 ECDSA algorithms: ecdsa-sha2-nistp256/384/521. These
    have been supported by OpenSSH since release 5.7.

To check whether a server is using the weak ssh-rsa public key
algorithm, for host authentication, try to connect to it after
removing the ssh-rsa algorithm from ssh(1)'s allowed list:

ssh -oHostKeyAlgorithms=-ssh-rsa user@host

If the host key verification fails and no other supported host key
types are available, the server software on that host should be
upgraded.

This release enables the UpdateHostKeys option by default to assist
the client by automatically migrating to better algorithms.

[1] “SHA-1 is a Shambles: First Chosen-Prefix Collision on SHA-1 and
Application to the PGP Web of Trust” Leurent, G and Peyrin, T
(2020) https://eprint.iacr.org/2020/014.pdf

I was going to check/test find a workaround on Fedora 33 only they still use OpenSSH 8.4p1. Once the Gitlab server is upgraded with an OpenSSH version that supports your client version I expect all will be good.

The other alternative is generate an SSH id-rsa key with SHA2 signature and put this in your gitlab profile, since SSH since 7.3 supports this. Running ssh -vvv git@gitlab.com shows that it runs OpenSSH 7.9 so it will be supported to solve it that way I expect.

Unfortunately OpenSSH 8.5p1 has only just been released so it’s unlikely to be available in most stable distros like Debian/Ubuntu/RHEL for quite some time yet - which means updating the server-side (gitlab.com) isn’t going to happen immediately. And since it’s just been released, other than the info above, there is no other workaround that has been posted yet. Will appear soon I’m sure, this is similar to when DSA keys were deprecated, and from the SSH client, or on the server-side had to add extra parameters to the config to allow it to accept the usage if you didn’t want to generate a new key. I had this and had to move from DSA to RSA keys since DSA then was far less secure apparently.

ED2519 signature as far as I know and still hit by this error. The ssh -o UpdateHostKeys=no -Tv git@gitlab.com will fix this (temporarily) which is not what you refer to. I think this is compatibility issue either on Gitlab or openSSH part.

@mezan I just use ssh -Tv and it works fine. I can also just do ssh -T (so it doesn’t print debug info), and a ED25519 key works perfectly well. So I expect rather it’s a problem at your end, rather than gitlab. Results from using -Tv parameter which shows the debug info and the ED25519 key that I am using to prove that it works fine.

ssh -Tv git@gitlab.com
OpenSSH_8.2p1 Ubuntu-4ubuntu0.2, OpenSSL 1.1.1f  31 Mar 2020
debug1: Reading configuration data /home/ian/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug1: Connecting to gitlab.com [172.65.251.78] port 22.
debug1: Connection established.
debug1: identity file /home/ian/.ssh/id_rsa type 0
debug1: identity file /home/ian/.ssh/id_rsa-cert type -1
debug1: identity file /home/ian/.ssh/id_dsa type -1
debug1: identity file /home/ian/.ssh/id_dsa-cert type -1
debug1: identity file /home/ian/.ssh/id_ecdsa type -1
debug1: identity file /home/ian/.ssh/id_ecdsa-cert type -1
debug1: identity file /home/ian/.ssh/id_ecdsa_sk type 10
debug1: identity file /home/ian/.ssh/id_ecdsa_sk-cert type -1
debug1: identity file /home/ian/.ssh/id_ed25519 type 3
debug1: identity file /home/ian/.ssh/id_ed25519-cert type -1
debug1: identity file /home/ian/.ssh/id_ed25519_sk type 12
debug1: identity file /home/ian/.ssh/id_ed25519_sk-cert type -1
debug1: identity file /home/ian/.ssh/id_xmss type -1
debug1: identity file /home/ian/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.2
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.9p1 Debian-10+deb10u2
debug1: match: OpenSSH_7.9p1 Debian-10+deb10u2 pat OpenSSH* compat 0x04000000
debug1: Authenticating to gitlab.com:22 as 'git'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:HbW3g8zUjNSksFbqTiUWPWg2Bq1x8xdGUrliXFzSnUw
debug1: Host 'gitlab.com' is known and matches the ECDSA host key.
debug1: Found key in /home/ian/.ssh/known_hosts:179
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey in after 134217728 blocks
debug1: Will attempt key: /home/ian/.ssh/id_rsa RSA SHA256:SfYvAdjD6qL47kIEl3D2cvStsqAJfxGcKNdyX51CIls
debug1: Will attempt key: /home/ian/.ssh/id_dsa 
debug1: Will attempt key: /home/ian/.ssh/id_ecdsa 
debug1: Will attempt key: /home/ian/.ssh/id_ecdsa_sk ECDSA-SK SHA256:7ptCI9i5iPfcWvLaZUYVPHryKJsc3BXAMdwCDc+FcxE authenticator
debug1: Will attempt key: /home/ian/.ssh/id_ed25519 ED25519 SHA256:TQ3MkyXE+HgaRO39U/QQJWiRB9MjxBu4X1KLbKh33hM
debug1: Will attempt key: /home/ian/.ssh/id_ed25519_sk ED25519-SK SHA256:gqCVSqTA7BRG7qdwglBdKFd50oSO/gjYbMK0SXUsUW0 authenticator
debug1: Will attempt key: /home/ian/.ssh/id_xmss 
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,keyboard-interactive
debug1: Next authentication method: publickey
debug1: Offering public key: /home/ian/.ssh/id_rsa RSA SHA256:SfYvAdjD6qL47kIEl3D2cvStsqAJfxGcKNdyX51CIls
debug1: Authentications that can continue: publickey,keyboard-interactive
debug1: Trying private key: /home/ian/.ssh/id_dsa
debug1: Trying private key: /home/ian/.ssh/id_ecdsa
debug1: Offering public key: /home/ian/.ssh/id_ecdsa_sk ECDSA-SK SHA256:7ptCI9i5iPfcWvLaZUYVPHryKJsc3BXAMdwCDc+FcxE authenticator
debug1: Authentications that can continue: publickey,keyboard-interactive
debug1: Offering public key: /home/ian/.ssh/id_ed25519 ED25519 SHA256:TQ3MkyXE+HgaRO39U/QQJWiRB9MjxBu4X1KLbKh33hM
debug1: Server accepts key: /home/ian/.ssh/id_ed25519 ED25519 SHA256:TQ3MkyXE+HgaRO39U/QQJWiRB9MjxBu4X1KLbKh33hM
debug1: Authentication succeeded (publickey).
Authenticated to gitlab.com ([172.65.251.78]:22).
debug1: channel 0: new [client-session]
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.
debug1: pledge: network
debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0
debug1: Remote: /authorized_keys %u %k:1: key options: command user-rc
debug1: Remote: /authorized_keys %u %k:1: key options: command user-rc
debug1: Sending environment.
debug1: Sending env LC_ADDRESS = en_GB.UTF-8
debug1: Sending env LC_NAME = en_GB.UTF-8
debug1: Sending env LC_MONETARY = en_GB.UTF-8
debug1: Sending env LC_PAPER = en_GB.UTF-8
debug1: Sending env LANG = en_GB.UTF-8
debug1: Sending env LC_IDENTIFICATION = en_GB.UTF-8
debug1: Sending env LC_TELEPHONE = en_GB.UTF-8
debug1: Sending env LC_MEASUREMENT = en_GB.UTF-8
debug1: Sending env LC_TIME = en_GB.UTF-8
debug1: Sending env LC_NUMERIC = en_GB.UTF-8
Welcome to GitLab, @iwalker!
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug1: client_input_channel_req: channel 0 rtype eow@openssh.com reply 0
debug1: channel 0: free: client-session, nchannels 1
Transferred: sent 3268, received 2664 bytes, in 0.4 seconds
Bytes per second: sent 7793.0, received 6352.7
debug1: Exit status 0

Gitlab Docs also verify the same commands that I use which work: GitLab and SSH keys | GitLab

If it was a problem with gitlab, then it wouldn’t work for me either.

I faced this issue, and after some search, I found out this topic.
I’m not so sure the issue is from our side.

Here’s a full debug output to help on this issue
from a freshly installed Fedora 34


$ ssh -Tv git@gitlab.com
OpenSSH_8.5p1, OpenSSL 1.1.1k  FIPS 25 Mar 2021
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Reading configuration data /etc/ssh/ssh_config.d/50-redhat.conf
debug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config
debug1: configuration requests final Match pass
debug1: re-parsing configuration
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Reading configuration data /etc/ssh/ssh_config.d/50-redhat.conf
debug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config
debug1: Connecting to gitlab.com [172.65.251.78] port 22.
debug1: Connection established.
debug1: identity file /home/myuser/.ssh/id_rsa type -1
debug1: identity file /home/myuser/.ssh/id_rsa-cert type -1
debug1: identity file /home/myuser/.ssh/id_dsa type -1
debug1: identity file /home/myuser/.ssh/id_dsa-cert type -1
debug1: identity file /home/myuser/.ssh/id_ecdsa type -1
debug1: identity file /home/myuser/.ssh/id_ecdsa-cert type -1
debug1: identity file /home/myuser/.ssh/id_ecdsa_sk type -1
debug1: identity file /home/myuser/.ssh/id_ecdsa_sk-cert type -1
debug1: identity file /home/myuser/.ssh/id_ed25519 type -1
debug1: identity file /home/myuser/.ssh/id_ed25519-cert type -1
debug1: identity file /home/myuser/.ssh/id_ed25519_sk type -1
debug1: identity file /home/myuser/.ssh/id_ed25519_sk-cert type -1
debug1: identity file /home/myuser/.ssh/id_xmss type -1
debug1: identity file /home/myuser/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.5
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.9p1 Debian-10+deb10u2
debug1: compat_banner: match: OpenSSH_7.9p1 Debian-10+deb10u2 pat OpenSSH* compat 0x04000000
debug1: Authenticating to gitlab.com:22 as 'git'
debug1: load_hostkeys: fopen /home/myuser/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ssh-ed25519
debug1: kex: server->client cipher: aes256-gcm@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: aes256-gcm@openssh.com MAC: <implicit> compression: none
debug1: kex: curve25519-sha256 need=32 dh_need=32
debug1: kex: curve25519-sha256 need=32 dh_need=32
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: SSH2_MSG_KEX_ECDH_REPLY received
debug1: Server host key: ssh-ed25519 SHA256:eUXGGm1YGsMAS7vkcx6JOJdOGHPem5gQp4taiCfCLB8
debug1: load_hostkeys: fopen /home/myuser/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: Host 'gitlab.com' is known and matches the ED25519 host key.
debug1: Found key in /home/myuser/.ssh/known_hosts:3
debug1: rekey out after 4294967296 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey in after 4294967296 blocks
debug1: Will attempt key: MySSHKeyName ECDSA SHA256:keyfingerprintxxx agent
debug1: Will attempt key: /home/myuser/.ssh/id_rsa
debug1: Will attempt key: /home/myuser/.ssh/id_dsa
debug1: Will attempt key: /home/myuser/.ssh/id_ecdsa
debug1: Will attempt key: /home/myuser/.ssh/id_ecdsa_sk
debug1: Will attempt key: /home/myuser/.ssh/id_ed25519
debug1: Will attempt key: /home/myuser/.ssh/id_ed25519_sk
debug1: Will attempt key: /home/myuser/.ssh/id_xmss
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,keyboard-interactive
debug1: Next authentication method: publickey
debug1: Offering public key: MySSHKeyName ECDSA SHA256:keyfingerprintxxx agent
debug1: Server accepts key: MySSHKeyName ECDSA SHA256:keyfingerprintxxx agent
debug1: Authentication succeeded (publickey).
Authenticated to gitlab.com ([172.65.251.78]:22).
debug1: channel 0: new [client-session]
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.
debug1: pledge: filesystem full
debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0
debug1: client_input_hostkeys: searching /home/myuser/.ssh/known_hosts for gitlab.com / (none)
debug1: client_input_hostkeys: searching /home/myuser/.ssh/known_hosts2 for gitlab.com / (none)
debug1: client_input_hostkeys: hostkeys file /home/myuser/.ssh/known_hosts2 does not exist
debug1: check_old_keys_othernames: hostkeys file /home/myuser/.ssh/known_hosts2 does not exist
debug1: Remote: /authorized_keys %u %k:1: key options: command user-rc
debug1: Remote: /authorized_keys %u %k:1: key options: command user-rc
debug1: Sending environment.
debug1: channel 0: setting env LANG = "en_US.UTF-8"
client_global_hostkeys_private_confirm: server gave bad signature for RSA key 0
Welcome to GitLab, @BitLogiK!
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug1: client_input_channel_req: channel 0 rtype eow@openssh.com reply 0
debug1: channel 0: free: client-session, nchannels 1
Transferred: sent 2868, received 3116 bytes, in 0.4 seconds
Bytes per second: sent 7764.5, received 8436.0
debug1: Exit status 0

client_global_hostkeys_private_confirm: server gave bad signature for RSA key 0
is present, but doesn’t prevent to connect, just display as a kind of warning. Not sure what it means exactly, but it definitively requires to investigate on the matter. Looks like there a small mistake in the GitLab SSH server configuration, or at least incompatibility with newest OpenSSH clients. I don’t even know what’s the matter with RSA here, as there is normally no RSA involved, servers keys use Ed25519 and my auth keys are ECDSA-384. I think that maybe the server is sending a message at the beginning with an optional signature. And this RSA signature is reported wrong by the client. Or eventually OpenSSH client is kind of lost, and threw this message with no relation whatsoever with the reality.

FTR, the warning message is displayed whatever the auth key.
ssh -oHostKeyAlgorithms=-ssh-ed25519 -Tv git@gitlab.com
makes the message displayed also.
The message is displayed well after connection, authentication (server and client), but just at the end when the tty session is starting.

The official issue thread : openssh 8.5p1-1 (#323700) · Issues · GitLab.org / GitLab · GitLab