Git user broken after following upgrade path from 11.9.8-ce to 14.2.1-ce.0

Hey there,

Thanks for reading this. I followed the upgrade path here: Upgrading GitLab | GitLab and up front everything appears to be working, except all users are now prompted for a password. I added a new key to my user and despite that I’m still prompted.

I’ve verified perms I’m aware of look correct:

$ ls -la | grep .ssh
drwx------  2 git               git        4096 Aug 25 10:24 .ssh

$ ls -la .ssh
total 184
drwx------  2 git  git    4096 Aug 25 10:38 .
drwxr-xr-x 21 root root   4096 Aug 25 09:32 ..
-r--------  1 git  git     136 Aug 25 10:38 agent-gitlab.cyverse.org
-rw-------  1 git  git  156460 Aug 25 10:12 authorized_keys
-rw-r--r--  1 git  git       0 Aug 25 10:12 authorized_keys.lock
-rw-------  1 git  git    1675 Nov  1  2016 id_rsa
-rw-r--r--  1 git  git     404 Nov  1  2016 id_rsa.pub
-rw-r--r--  1 git  git    2605 Feb 19  2019 known_hosts

I’ve verified my key types are allowed.

I’ve done a gitlab-ctl reconfigure on it for good measure.

Following is the output of simply trying to ssh as git to the instance with the following command ssh git@gitlab.example.org -i /home/USER/.ssh/id_ecdsa -vvvv

OpenSSH_8.2p1 Ubuntu-4ubuntu0.3, OpenSSL 1.1.1f  31 Mar 2020
debug1: Reading configuration data /home/USER/.ssh/config
debug1: /home/USER/.ssh/config line 1: Applying options for *
debug1: /home/USER/.ssh/config line 22: Applying options for *.example.org
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug2: resolving "gitlab.example.org" port 2222
debug2: ssh_connect_direct
debug1: Connecting to gitlab.example.org [1.1.1.1] port 2222.
debug1: Connection established.
debug1: identity file /home/USER/.ssh/id_ecdsa type 2
debug1: identity file /home/USER/.ssh/id_ecdsa-cert type -1
debug1: identity file /home/USER/.ssh/id_rsa type 0
debug1: identity file /home/USER/.ssh/id_rsa-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.3
debug1: Remote protocol version 2.0, remote software version OpenSSH_8.2p1 Ubuntu-4ubuntu0.3
debug1: match: OpenSSH_8.2p1 Ubuntu-4ubuntu0.3 pat OpenSSH* compat 0x04000000
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to gitlab.example.org:2222 as 'git'
debug3: put_host_port: [gitlab.example.org]:2222
debug3: hostkeys_foreach: reading file "/home/USER/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /home/USER/.ssh/known_hosts:59
debug3: load_hostkeys: loaded 1 keys from [gitlab.example.org]:2222
debug3: order_hostkeyalgs: prefer hostkeyalgs: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,ext-info-c
debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ecdsa-sha2-nistp256-cert-v01@openssh.com,ssh-ed25519-cert-v01@openssh.com,sk-ssh-ed25519-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,sk-ecdsa-sha2-nistp256@openssh.com,ssh-ed25519,sk-ssh-ed25519@openssh.com,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com,zlib
debug2: compression stoc: none,zlib@openssh.com,zlib
debug2: languages ctos: 
debug2: languages stoc: 
debug2: first_kex_follows 0 
debug2: reserved 0 
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256
debug2: host key algorithms: rsa-sha2-512,rsa-sha2-256,ssh-rsa,ecdsa-sha2-nistp256,ssh-ed25519
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com
debug2: compression stoc: none,zlib@openssh.com
debug2: languages ctos: 
debug2: languages stoc: 
debug2: first_kex_follows 0 
debug2: reserved 0 
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: Server host key: ecdsa-sha2-nistp256 OMITTED
debug3: put_host_port: [1.1.1.1]:2222
debug3: put_host_port: [gitlab.example.org]:2222
debug3: hostkeys_foreach: reading file "/home/USER/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /home/USER/.ssh/known_hosts:59
debug3: load_hostkeys: loaded 1 keys from [gitlab.example.org]:2222
debug3: hostkeys_foreach: reading file "/home/USER/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /home/USER/.ssh/known_hosts:60
debug3: load_hostkeys: loaded 1 keys from [1.1.1.1]:2222
debug1: Host '[gitlab.example.org]:2222' is known and matches the ECDSA host key.
debug1: Found key in /home/USER/.ssh/known_hosts:59
debug3: send packet: type 21
debug2: set_newkeys: mode 1
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug1: SSH2_MSG_NEWKEYS received
debug2: set_newkeys: mode 0
debug1: rekey in after 134217728 blocks
debug1: Will attempt key: /home/USER/.ssh/id_rsa RSA SHA256:OMITTED explicit agent
debug1: Will attempt key: /home/USER/.ssh/id_ecdsa ECDSA SHA256:OMITTED explicit
debug2: pubkey_prepare: done
debug3: send packet: type 5
debug3: receive packet: type 7
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,sk-ssh-ed25519@openssh.com,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ecdsa-sha2-nistp256@openssh.com>
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 53
debug3: input_userauth_banner




************************************************
OMMITTED BANNER TEXT
************************************************



debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,password
debug3: start over, passed a different list publickey,password
debug3: preferred gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering public key: /home/USER/.ssh/id_rsa RSA SHA256:OMITTED explicit agent
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,password
debug1: Offering public key: /home/USER/.ssh/id_ecdsa ECDSA SHA256:OMITTED explicit
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,password
debug2: we did not send a packet, disable method
debug3: authmethod_lookup password
debug3: remaining preferred: ,password
debug3: authmethod_is_enabled password
debug1: Next authentication method: password
git@gitlab.example.org's password: 

I am able to ssh as other users onto the system.

Any and all advice is greatly appreciated.

As it seems is always the case. This was a server config issue and not a part of gitlab upgrades.

tail -f /var/log/auth.log showed me:

Aug 25 11:02:18 gitlabtest sshd[3053069]: User git from 3.3.3.3 not allowed because none of user's groups are listed in AllowGroups
Aug 25 11:02:19 gitlabtest sshd[3053069]: Connection closed by invalid user git 3.3.3.3 port 46376 [preauth]

This, despite having AllowGroups git in my /etc/ssh/sshd_config.d/example.conf file. It turns out that is overwritten by /etc/ssh/sshd_config, which had other groups, but not git

Thanks for making an awesome product and I hope this helps someone in the future.

2 Likes