Again a Permission denied question about ssh connection

Alcare · November 14, 2019, 1:16pm

Hello,
I’m really frustrated to experience a such common error, as far as I saw. I read tons of posts here and on SO about such problem but helped me.
My main problem is to not really understand where the problem could come from.

I’ve a self-hosted gitlab on a Ubuntu 18.04 VM.
I’ve AFAIK correctly set it up, i.e. sudo gitlab-rake gitlab:check --trace doesn’t reveal anything wrong. And I’m in a fully common configuration.

So I’ve created a user tech in gitlab and a test project.
I’ve set the git global config for name and email and then created the ssh key, copied the pub one, and paste it in the tech’s account. So both fingerprints match.

And I cannot clone it.

But first, ssh -T git@gitrepos.me-in.com seems to correctly find the rsa_key and send it but replies Permission denied (publickey,password).

So I’d like to understand where to look for the problem.
As far as I understood, when I execute this command there is nothing related with tech user?
It only concerns the ability of the common user git to connect to gitlab via ssh right?

So what do I have to verify to find who’s blocking or refusing something because according to ssh -Tvvv... I don’t understand if it doesn’t receive anything or if it receives an error.

me@me:~$ ssh -Tvvv git@gitrepos.me-in.com
OpenSSH_6.0p1 Debian-4+deb7u4, OpenSSL 1.0.1e 11 Feb 2013
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug2: ssh_connect: needpriv 0
debug1: Connecting to gitrepos.me-in.com [82.64.13.218] port 22.
debug1: Connection established.
debug3: Incorrect RSA1 identifier
debug3: Could not load "/home/me/.ssh/id_rsa" as a RSA1 public key
debug1: identity file /home/me/.ssh/id_rsa type 1
debug1: Checking blacklist file /usr/share/ssh/blacklist.RSA-4096
debug1: Checking blacklist file /etc/ssh/blacklist.RSA-4096
debug1: identity file /home/me/.ssh/id_rsa-cert type -1
debug3: Incorrect RSA1 identifier
debug3: Could not load "/home/me/.ssh/id_dsa" as a RSA1 public key
debug1: identity file /home/me/.ssh/id_dsa type 2
debug1: Checking blacklist file /usr/share/ssh/blacklist.DSA-1024
debug1: Checking blacklist file /etc/ssh/blacklist.DSA-1024
debug1: identity file /home/me/.ssh/id_dsa-cert type -1
debug1: identity file /home/me/.ssh/id_ecdsa type -1
debug1: identity file /home/me/.ssh/id_ecdsa-cert type -1
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.6p1 Ubuntu-4ubuntu0.3
debug1: match: OpenSSH_7.6p1 Ubuntu-4ubuntu0.3 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.0p1 Debian-4+deb7u4
debug2: fd 3 setting O_NONBLOCK
debug3: load_hostkeys: loading entries for host "gitrepos.me-in.com" from file "/home/me/.ssh/known_hosts"
debug3: load_hostkeys: found key type ECDSA in file /home/me/.ssh/known_hosts:11
debug3: load_hostkeys: loaded 1 keys
debug3: order_hostkeyalgs: prefer hostkeyalgs: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug2: kex_parse_kexinit: ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-rsa-cert-v01@openssh.com,ssh-dss-cert-v01@openssh.com,ssh-rsa-cert-v00@openssh.com,ssh-dss-cert-v00@openssh.com,ssh-rsa,ssh-dss
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-sha2-256,hmac-sha2-256-96,hmac-sha2-512,hmac-sha2-512-96,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-sha2-256,hmac-sha2-256-96,hmac-sha2-512,hmac-sha2-512-96,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit: 
debug2: kex_parse_kexinit: 
debug2: kex_parse_kexinit: first_kex_follows 0 
debug2: kex_parse_kexinit: reserved 0 
debug2: kex_parse_kexinit: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1
debug2: kex_parse_kexinit: ssh-rsa,rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519
debug2: kex_parse_kexinit: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: kex_parse_kexinit: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: kex_parse_kexinit: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: kex_parse_kexinit: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: kex_parse_kexinit: none,zlib@openssh.com
debug2: kex_parse_kexinit: none,zlib@openssh.com
debug2: kex_parse_kexinit: 
debug2: kex_parse_kexinit: 
debug2: kex_parse_kexinit: first_kex_follows 0 
debug2: kex_parse_kexinit: reserved 0 
debug2: mac_setup: found hmac-sha1
debug1: kex: server->client aes128-ctr hmac-sha1 none
debug2: mac_setup: found hmac-sha1
debug1: kex: client->server aes128-ctr hmac-sha1 none
debug1: sending SSH2_MSG_KEX_ECDH_INIT
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ECDSA 23:a1:a2:88:36:b8:ea:84:e9:29:b7:cb:0c:72:ac:23
debug3: load_hostkeys: loading entries for host "gitrepos.me-in.com" from file "/home/me/.ssh/known_hosts"
debug3: load_hostkeys: found key type ECDSA in file /home/me/.ssh/known_hosts:11
debug3: load_hostkeys: loaded 1 keys
debug3: load_hostkeys: loading entries for host "82.64.13.218" from file "/home/me/.ssh/known_hosts"
debug3: load_hostkeys: found key type ECDSA in file /home/me/.ssh/known_hosts:13
debug3: load_hostkeys: loaded 1 keys
debug1: Host 'gitrepos.me-in.com' is known and matches the ECDSA host key.
debug1: Found key in /home/me/.ssh/known_hosts:11
debug1: ssh_ecdsa_verify: signature correct
debug2: kex_derive_keys
debug2: set_newkeys: mode 1
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug2: set_newkeys: mode 0
debug1: SSH2_MSG_NEWKEYS received
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug2: key: /home/me/.ssh/id_dsa (0xb82f5040)
debug2: key: /home/me/.ssh/id_rsa (0xb82eed80)
debug2: key: /home/me/.ssh/id_ecdsa ((nil))
debug1: Authentications that can continue: publickey,password
debug3: start over, passed a different list publickey,password
debug3: preferred gssapi-keyex,gssapi-with-mic,publickey,keyboard-interactive
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering DSA public key: /home/me/.ssh/id_dsa
debug3: send_pubkey_test
debug2: we sent a publickey packet, wait for reply
debug1: Authentications that can continue: publickey,password
debug1: Offering RSA public key: /home/me/.ssh/id_rsa
debug3: send_pubkey_test
debug2: we sent a publickey packet, wait for reply
debug1: Authentications that can continue: publickey,password
debug1: Trying private key: /home/me/.ssh/id_ecdsa
debug3: no such identity: /home/me/.ssh/id_ecdsa
debug2: we did not send a packet, disable method
debug1: No more authentication methods to try.
Permission denied (publickey,password).

paramveer.singh · November 18, 2019, 6:59am

Hi Alcare!

There are a couple of things I’d like to know:

Can you share the command you have used to create the SSH key pair? Also, can you also share the ssh_config file from ‘/etc/ssh/’?
Are you copying the public-key onto GitLab or the private key?

Thanks

Alcare · November 18, 2019, 7:45am

Hello,

Except if there is something I didn’t understand, I followed the doc but I guess the problem comes form the server where Gitlab runs as I get the same problem whatever the client machine I use. I can access to this server via ssh sucessfully (by login/password, I didn’t try via ssh key), I can also access Gitlab via https, but the problem is to access Gitlab via ssh.

To generate the key, I used the following command and yes, I copied the public key.

ssh-keygen -t rsa -b 4096

I also tried:

ssh-keygen -t ed25519

The server’s ssh config is:

Host *                                                                                                                                                                        
#   ForwardAgent no                                                                                                                                                           
#   ForwardX11 no                                                                                                                                                             
#   ForwardX11Trusted yes                                                                                                                                                     
#   PasswordAuthentication yes                                                                                                                                                
#   HostbasedAuthentication no                                                                                                                                                
#   GSSAPIAuthentication no                                                                                                                                                   
#   GSSAPIDelegateCredentials no                                                                                                                                              
#   GSSAPIKeyExchange no                                                                                                                                                      
#   GSSAPITrustDNS no                                                                                                                                                         
#   BatchMode no                                                                                                                                                              
#   CheckHostIP yes                                                                                                                                                           
#   AddressFamily any                                                                                                                                                         
#   ConnectTimeout 0                                                                                                                                                          
#   StrictHostKeyChecking ask                                                                                                                                                 
#   IdentityFile ~/.ssh/id_rsa                                                                                                                                                
#   IdentityFile ~/.ssh/id_dsa                                                                                                                                                
#   IdentityFile ~/.ssh/id_ecdsa                                                                                                                                              
#   IdentityFile ~/.ssh/id_ed25519                                                                                                                                           
#   Port 22                                                                                                                                                                   
#   Protocol 2                                                                                                                                                                
#   Ciphers aes128-ctr,aes192-ctr,aes256-ctr,aes128-cbc,3des-cbc                                                                                                              
#   MACs hmac-md5,hmac-sha1,umac-64@openssh.com                                                                                                                               
#   EscapeChar ~                                                                                                                                                              
#   Tunnel no                                                                                                                                                                 
#   TunnelDevice any:any                                                                                                                                                      
#   PermitLocalCommand no                                                                                                                                                     
#   VisualHostKey no                                                                                                                                                          
#   ProxyCommand ssh -q -W %h:%p gateway.example.com                                                                                                                          
#   RekeyLimit 1G 1h                                                                                                                                                              
SendEnv LANG LC_*                                                                                                                                                             
HashKnownHosts yes
    GSSAPIAuthentication yes

In fact, I got the following message at the first connection:
The authenticity of host ‘xxxxxxx’ can’t be established.

But I don’t really know if it means that ssh connection is granted by the server and if the problem comes from Gitlab which rejects the request.
Regards,

paramveer.singh · November 18, 2019, 8:05am

Hi!

I think there are a few things you can check here:

View the ‘/etc/hosts’ file and check if your server’s name is correctly mapped with the IPAddress of that server.
You can try deleting the directory “~/.ssh/” and ‘yum remove’ the ssh and install it again.
One of the reasons could be that since you have generated multiple keys on your server, the system must be picking up a wrong key, so delete the existing keys and create a new key, copy that key to GitLab>SSH-keys and make the following changes in the ‘/etc/ssh/ssh_config’ file:

 Host gitrepos.me-in.com
   Preferredauthentications publickey
   IdentityFile ~/.ssh/id_rsa

where “gitrepos.me-in.com” is the name of your host machine as mapped in ‘/etc/hosts’ file.

Let me know, if anything works out or what error you might face after doing the above changes!

Alcare · November 18, 2019, 8:30am

information in /etc/hosts is correct.

About these points 2 and 3, I have a doubt about your sentence:

since you have generated multiple keys on your server

Could you be more precise ?
I generated the keys on the client, and then copied them on the server’s Gitlab.
Do you mean I had to generate these keys on the server side?

Anyway about point 2, as I have the same problem on 3 clients, 2 Linux and one Windows, I think the problem is more on the server than on clients no?

paramveer.singh · November 18, 2019, 8:43am

On the client side, a user can create multiple SSH-keys, using different algorithms, like a key using ed25519 and a key using RSA. So, in order for the system to decide which one to use, we have to mention it in the ‘/etc/ssh/ssh_config’ file:

 Host gitrepos.me-in.com
   Preferredauthentications publickey
   IdentityFile ~/.ssh/id_rsa

So, try removing the directory ‘~/.ssh/’, then re-installing ‘ssh’, and create only one key and then make the above changes in /etc/ssh/ssh_config’ file and copy ‘id_rsa.pub’ or ‘id_ed25519’ file’s contents in your GitLab account under ‘SSH Keys’ and then run the following command:

ssh -T git@gitlab-host-server

Alcare · November 18, 2019, 9:13am

Ohh cool, it works now.
So the client certainly didn’t deal with the right ssh keys.

thanks a lot!!!

axbender · July 3, 2020, 4:46pm

I got the same prob here. Tried - like Alcare - different key types, to no avail. My .ssh/config contains the following entries (and I can see - using -Tvvv - that the correct key file is picked up and the correct fingerprint is transferred):

ServerAliveInterval 120
ServerAliveCountMax 5

Host i40gitlab gitlab.i40.ifm-datalink.net
HostName gitlab.i40.ifm-datalink.net
IdentityFile ~/.ssh/i40gitlab/id_ed25519x
User git

I keep getting this error (last lines):
#########################
debug1: Will attempt key: /root/.ssh/i40gitlab/id_ed25519 ED25519 SHA256:awkqAeF1wg73u50Z+qIl2/awHkgi25PJnJNS5YORRjM explicit
debug2: pubkey_prepare: done
debug3: send packet: type 5
debug3: receive packet: type 7
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521>
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey
debug3: start over, passed a different list publickey
debug3: preferred gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering public key: /root/.ssh/i40gitlab/id_ed25519 ED25519 SHA256:awkqAeF1wg73u50Z+qIl2/awHkgi25PJnJNS5YORRjM explicit
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey
debug2: we did not send a packet, disable method
debug1: No more authentication methods to try.
git@gitlab.i40.ifm-datalink.net: Permission denied (publickey).
######################

Any clues?

Topic		Replies	Views
Gitlab.com does not accept ssh keys, resulting in "Permission denied" Self-managed ssh	2	5350	January 26, 2021
Access via SSH stop working: Permission denied (publickey) How to Use GitLab permissions	4	13049	June 14, 2022
Git issue : "Permission denied (publickey,keyboard-interactive)" with "ssh -T git@gitlab com" command How to Use GitLab issues , ssh	0	2186	March 19, 2022
Permission denied (publickey) while using SSH connection How to Use GitLab	2	5802	November 14, 2019
Help using SSH to log in to gitlab Self-managed	4	781	October 17, 2021

Again a Permission denied question about ssh connection

Related topics