I have a user reporting trouble with interacting with our GitLab server after they were able to use it successfully for several month. Essentially they’re getting this from ssh when attempting to execute git “Permission denied, please try again.” The only differential appears to be that they changed their processor in their workstation.
No such problems happen for them on GitHub or other git servers, so they’re thinking it could be server related. Unfortunately also no other users are complaining of the issue.
As simple reproduction of the failure is to execute git ls-remote:
ssh -vv -p 43210 -A username@hostname1.tld 'git ls-remote git@gitlab.hostname2.tld:projects/project_name.git'
It gets onto the server (hostname1.tld) just fine (for example, if they issue ls instead of the git ls-remote command there the remote server directory contents are returned). However, when it executes the git ls-remote command, we see the following:
...
debug1: Authentication succeeded (publickey).
Authenticated to hostname1.tld ([hostname1.tld]:43210).
debug1: setting up multiplex master socket
debug2: fd 4 setting O_NONBLOCK
debug1: channel 0: new [/tmp/username@hostname1.tld:43210]
debug1: channel 1: new [client-session]
debug2: channel 1: send open
debug1: Entering interactive session.
debug1: pledge: id
debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0
debug2: channel_input_open_confirmation: channel 1: callback start
debug1: Requesting authentication agent forwarding.
debug2: channel 1: request auth-agent-req@openssh.com confirm 0
debug2: fd 3 setting TCP_NODELAY
debug2: client_session2_setup: id 1
debug1: Sending environment.
debug1: Sending env LANG = en_US.UTF-8
debug2: channel 1: request env confirm 0
debug1: Sending env LC_TERMINAL = iTerm2
debug2: channel 1: request env confirm 0
debug1: Sending env LC_TERMINAL_VERSION = 3.3
debug2: channel 1: request env confirm 0
debug1: Sending env LC_CTYPE = en_US.UTF-8
debug2: channel 1: request env confirm 0
debug1: Sending command: git ls-remote git@gitlab.hostname2.tld:projects/project_name.git
debug2: channel 1: request exec confirm 1
debug2: channel_input_open_confirmation: channel 1: callback done
debug2: channel 1: open confirm rwindow 0 rmax 32768
debug2: channel 1: rcvd adjust 2097152
debug2: channel_input_status_confirm: type 99 id 1
debug2: exec request accepted on channel 1
debug1: client_input_channel_open: ctype auth-agent@openssh.com rchan 2 win 65536 max 16384
debug2: fd 8 setting O_NONBLOCK
debug1: channel 2: new [authentication agent connection]
debug1: confirm auth-agent@openssh.com
debug2: channel 2: rcvd eof
debug2: channel 2: output open -> drain
debug2: channel 2: obuf empty
debug2: channel 2: chan_shutdown_write (i0 o1 sock 8 wfd 8 efd -1 [closed])
debug2: channel 2: output drain -> closed
debug1: channel 2: FORCE input drain
debug2: channel 2: ibuf empty
debug2: channel 2: send eof
debug2: channel 2: input drain -> closed
debug2: channel 2: send close
debug2: channel 1: rcvd ext data 38
Permission denied, please try again.
debug2: channel 1: written 38 to efd 7
debug2: channel 1: rcvd ext data 38
Permission denied, please try again.
debug2: channel 1: written 38 to efd 7
debug2: channel 1: rcvd ext data 41
Permission denied (publickey,password).
debug2: channel 1: written 41 to efd 7
debug1: client_input_channel_req: channel 1 rtype exit-status reply 0
debug1: client_input_channel_req: channel 1 rtype eow@openssh.com reply 0
debug2: channel 1: rcvd eow
debug2: channel 1: chan_shutdown_read (i0 o0 sock -1 wfd 5 efd 7 [write])
debug2: channel 1: input open -> closed
debug2: channel 1: rcvd ext data 126
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
On the GitLab server itself (hostname2.tld) with ssh logging turned up to DEBUG, the following can be seen in the auth.log:
gitlab sshd[8053]: debug1: Forked child 8075.
gitlab sshd[8075]: debug1: Set /proc/self/oom_score_adj to 0
gitlab sshd[8075]: debug1: rexec start in 5 out 5 newsock 5 pipe 7 sock 8
gitlab sshd[8075]: debug1: inetd sockets after dupping: 3, 3
gitlab sshd[8075]: Connection from hostname1.tld port 38412 on hostname2.tld port 22
gitlab sshd[8075]: debug1: Client protocol version 2.0; client software version OpenSSH_7.4
gitlab sshd[8075]: debug1: match: OpenSSH_7.4 pat OpenSSH* compat 0x04000000
gitlab sshd[8075]: debug1: Enabling compatibility mode for protocol 2.0
gitlab sshd[8075]: debug1: Local version string SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.8
gitlab sshd[8075]: debug1: permanently_set_uid: 108/65534 [preauth]
gitlab sshd[8075]: debug1: list_hostkey_types: ssh-rsa,rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519 [preauth]
gitlab sshd[8075]: debug1: SSH2_MSG_KEXINIT sent [preauth]
gitlab sshd[8075]: debug1: SSH2_MSG_KEXINIT received [preauth]
gitlab sshd[8075]: debug1: kex: algorithm: curve25519-sha256@libssh.org [preauth]
gitlab sshd[8075]: debug1: kex: host key algorithm: ecdsa-sha2-nistp256 [preauth]
gitlab sshd[8075]: debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none [preauth]
gitlab sshd[8075]: debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none [preauth]
gitlab sshd[8075]: debug1: expecting SSH2_MSG_KEX_ECDH_INIT [preauth]
gitlab sshd[8075]: debug1: rekey after 134217728 blocks [preauth]
gitlab sshd[8075]: debug1: SSH2_MSG_NEWKEYS sent [preauth]
gitlab sshd[8075]: debug1: expecting SSH2_MSG_NEWKEYS [preauth]
gitlab sshd[8075]: debug1: SSH2_MSG_NEWKEYS received [preauth]
gitlab sshd[8075]: debug1: rekey after 134217728 blocks [preauth]
gitlab sshd[8075]: debug1: KEX done [preauth]
gitlab sshd[8075]: debug1: userauth-request for user git service ssh-connection method none [preauth]
gitlab sshd[8075]: debug1: attempt 0 failures 0 [preauth]
gitlab sshd[8075]: debug1: PAM: initializing for "git"
gitlab sshd[8075]: debug1: PAM: setting PAM_RHOST to "hostname1.tld"
gitlab sshd[8075]: debug1: PAM: setting PAM_TTY to "ssh"
gitlab sshd[8075]: debug1: userauth-request for user git service ssh-connection method password [preauth]
gitlab sshd[8075]: debug1: attempt 1 failures 0 [preauth]
gitlab sshd[8075]: Failed none for git from hostname1.tld port 38412 ssh2
gitlab sshd[8075]: debug1: userauth-request for user git service ssh-connection method password [preauth]
gitlab sshd[8075]: debug1: attempt 2 failures 1 [preauth]
gitlab sshd[8075]: Failed password for git from hostname1.tld port 38412 ssh2
gitlab sshd[8075]: debug1: userauth-request for user git service ssh-connection method password [preauth]
gitlab sshd[8075]: debug1: attempt 3 failures 2 [preauth]
gitlab sshd[8075]: Failed password for git from hostname1.tld port 38412 ssh2
gitlab sshd[8075]: Connection closed by hostname1.tld port 38412 [preauth]
gitlab sshd[8075]: debug1: do_cleanup [preauth]
gitlab sshd[8075]: debug1: monitor_read_log: child log fd closed
gitlab sshd[8075]: debug1: do_cleanup
gitlab sshd[8075]: debug1: PAM: cleanup
gitlab sshd[8075]: debug1: Killing privsep child 8076
gitlab sshd[8075]: debug1: audit_event: unhandled event 12
It appears that it’s failing at this error above: Failed password for git
It seems as though the key they have set within GitLab from their workstation is not passing on through the two ssh connections. Are there any methodologies to troubleshoot this further?