Cannot git ssh on FQDN but working on IP ADDRESS

I am currently stuck on an error in git using gitlab 14.10.4. I can ssh -T git@IP_ADDRESS but not ssh -T git@FQDN . I am using the same key, same machine where I can connect to git via ssh on ip address. What am I missing?

debug1: Will attempt key: /home/user/.ssh/id_rsa RSA SHA256:VX3JDVrZYNtFpFUhiQR11IYdRCotA/yl/H0DodwKqRY agent
debug2: pubkey_prepare: done
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521>
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,password
debug1: Next authentication method: publickey
debug1: Offering public key: /home/user/.ssh/id_rsa RSA SHA256:VX3JDVrZYNtFpFUhiQR11IYdRCotA/yl/H0DodwKqRY agent
debug2: we sent a publickey packet, wait for reply
debug1: Authentications that can continue: publickey,password
debug2: we did not send a packet, disable method
debug1: No more authentication methods to try.
git@fqdn: Permission denied (publickey,password).

my setup would be gitlab server is proxied on a separate web server. but I already tried attaching the ip address directly on the gitlab server and still encounter the same error where I can connect to the gitlab server via ssh on ip address, while not via FQDN.

Do you have some configuration for the ip (or the fqdn) in your .ssh/config?

Hi, here is my ssh_config, am I missing something here?

Host FQDN
PasswordAuthentication no
IdentityFile ~/.ssh/id_rsa
CheckHostIP yes

I might be wrong, but when you have IdentityFile ~/.ssh/id_rsa in your ssh config, I think ssh only tries that (and those that are provided by an agent if you have one running). That suggests that it uses a different key when you ssh to the ip, try to show us the output like the above, but from an attempt to connect to the ip.

But as ~/.ssh/id_rsa is (part of) the default, the explanation must lie in the “part of”, as an ssh to the ip wouldn’t try other keys (unless you didn’t show us your entire .ssh/config, there might be default settings).

ssh -Tvvv git@ip_address

OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n  7 Dec 2017
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug2: resolving "ip_address" port 22
debug2: ssh_connect_direct: needpriv 0
debug1: Connecting to "ip_address" ["ip_address"] port 22.
debug1: Connection established.
debug1: identity file /home/user/.ssh/id_rsa type 0
debug1: key_load_public: No such file or directory
debug1: identity file /home/user/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/user/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/user/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/user/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/user/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/user/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/user/.ssh/id_ed25519-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.3
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.4
debug1: match: OpenSSH_7.4 pat OpenSSH* compat 0x04000000
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to ip_address:22 as 'git'
debug3: hostkeys_foreach: reading file "/home/user/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /home/user/.ssh/known_hosts:105
debug3: load_hostkeys: loaded 1 keys from ip_address
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:nUDg7AX2VhVP0548lunrdeLsAXDuSL6nXnYpsGD01QI
debug3: hostkeys_foreach: reading file "/home/user/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /home/user/.ssh/known_hosts:105
debug3: load_hostkeys: loaded 1 keys from ip_address
debug1: Host 'ip_address' is known and matches the ECDSA host key.
debug1: Found key in /home/user/.ssh/known_hosts:105
debug3: send packet: type 21
debug2: set_newkeys: mode 1
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug1: SSH2_MSG_NEWKEYS received
debug2: set_newkeys: mode 0
debug1: rekey after 134217728 blocks
debug2: key: /home/user/.ssh/id_rsa (0x7fffc211eb30)
debug2: key: /home/user/.ssh/id_dsa ((nil))
debug2: key: /home/user/.ssh/id_ecdsa ((nil))
debug2: key: /home/user/.ssh/id_ed25519 ((nil))
debug3: send packet: type 5
debug3: receive packet: type 7
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<rsa-sha2-256,rsa-sha2-512>
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic
debug3: start over, passed a different list publickey,gssapi-keyex,gssapi-with-mic
debug3: preferred gssapi-keyex,gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup gssapi-keyex
debug3: remaining preferred: gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_is_enabled gssapi-keyex
debug1: Next authentication method: gssapi-keyex
debug1: No valid Key exchange context
debug2: we did not send a packet, disable method
debug3: authmethod_lookup gssapi-with-mic
debug3: remaining preferred: publickey,keyboard-interactive,password
debug3: authmethod_is_enabled gssapi-with-mic
debug1: Next authentication method: gssapi-with-mic
debug1: Unspecified GSS failure.  Minor code may provide more information
No Kerberos credentials available (default cache: FILE:/tmp/krb5cc_1000)

debug1: Unspecified GSS failure.  Minor code may provide more information
No Kerberos credentials available (default cache: FILE:/tmp/krb5cc_1000)

debug2: we did not send a packet, disable method
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering public key: RSA SHA256:GCOmXVN7463SEXRS+2MqMPh9l6X/+2J1ZvGHCm/6aiQ /home/user/.ssh/id_rsa
debug3: send_pubkey_test
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 60
debug1: Server accepts key: pkalg rsa-sha2-512 blen 279
debug2: input_userauth_pk_ok: fp SHA256:GCOmXVN7463SEXRS+2MqMPh9l6X/+2J1ZvGHCm/6aiQ
debug3: sign_and_send_pubkey: RSA SHA256:GCOmXVN7463SEXRS+2MqMPh9l6X/+2J1ZvGHCm/6aiQ
debug3: send packet: type 50
debug3: receive packet: type 52
debug1: Authentication succeeded (publickey).
Authenticated to ip_address ([ip_address]:22).
debug1: channel 0: new [client-session]
debug3: ssh_session2_open: channel_new: 0
debug2: channel 0: send open
debug3: send packet: type 90
debug1: Requesting no-more-sessions
debug3: send packet: type 80
debug1: Entering interactive session.
debug1: pledge: network
debug3: receive packet: type 80
debug1: client_input_global_request: rtype want_reply 0
debug3: receive packet: type 4
debug1: Remote: Forced command.
debug3: receive packet: type 4
debug1: Remote: Port forwarding disabled.
debug3: receive packet: type 4
debug1: Remote: X11 forwarding disabled.
debug3: receive packet: type 4
debug1: Remote: Agent forwarding disabled.
debug3: receive packet: type 4
debug1: Remote: PTY allocation disabled.
debug3: receive packet: type 4
debug1: Remote: Forced command.
debug3: receive packet: type 4
debug1: Remote: Port forwarding disabled.
debug3: receive packet: type 4
debug1: Remote: X11 forwarding disabled.
debug3: receive packet: type 4
debug1: Remote: Agent forwarding disabled.
debug3: receive packet: type 4
debug1: Remote: PTY allocation disabled.
debug3: receive packet: type 91
debug2: channel_input_open_confirmation: channel 0: callback start
debug2: fd 3 setting TCP_NODELAY
debug3: ssh_packet_set_tos: set IP_TOS 0x08
debug2: client_session2_setup: id 0
debug1: Sending environment.
debug3: Ignored env LS_COLORS
debug3: Ignored env HOSTTYPE
debug3: Ignored env LESSCLOSE
debug1: Sending env LANG = C.UTF-8
debug2: channel 0: request env confirm 0
debug3: send packet: type 98
debug3: Ignored env WSL_DISTRO_NAME
debug3: Ignored env USER
debug3: Ignored env PWD
debug3: Ignored env HOME
debug3: Ignored env NAME
debug3: Ignored env XDG_DATA_DIRS
debug3: Ignored env SHELL
debug3: Ignored env TERM
debug3: Ignored env SHLVL
debug3: Ignored env LOGNAME
debug3: Ignored env PATH
debug3: Ignored env WSLENV
debug3: Ignored env LESSOPEN
debug3: Ignored env _
debug2: channel 0: request shell confirm 1
debug3: send packet: type 98
debug2: channel_input_open_confirmation: channel 0: callback done
debug2: channel 0: open confirm rwindow 0 rmax 32768
debug2: channel 0: rcvd adjust 2097152
debug3: receive packet: type 99
debug2: channel_input_status_confirm: type 99 id 0
debug2: shell request accepted on channel 0
Welcome to GitLab, @username!
debug3: receive packet: type 98
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug3: receive packet: type 98
debug1: client_input_channel_req: channel 0 rtype reply 0
debug2: channel 0: rcvd eow
debug2: channel 0: close_read
debug2: channel 0: input open -> closed
debug3: receive packet: type 96
debug2: channel 0: rcvd eof
debug2: channel 0: output open -> drain
debug2: channel 0: obuf empty
debug2: channel 0: close_write
debug2: channel 0: output drain -> closed
debug3: receive packet: type 97
debug2: channel 0: rcvd close
debug3: channel 0: will not send data after close
debug2: channel 0: almost dead
debug2: channel 0: gc: notify user
debug2: channel 0: gc: user detached
debug2: channel 0: send close
debug3: send packet: type 97
debug2: channel 0: is dead
debug2: channel 0: garbage collecting
debug1: channel 0: free: client-session, nchannels 1
debug3: channel 0: status: The following connections are open:
  #0 client-session (t4 r0 i3/0 o3/0 fd -1/-1 cc -1)

debug3: send packet: type 1
Transferred: sent 2688, received 3200 bytes, in 0.4 seconds
Bytes per second: sent 6217.7, received 7402.0
debug1: Exit status 0
here is the entire ssh_config

*# This is the ssh client system-wide configuration file.  See*
*# ssh_config(5) for more information.  This file provides defaults for*
*# users, and the values can be changed in per-user configuration files*
*# or on the command line.*

*# Configuration data is parsed as follows:*
*#  1. command line options*
*#  2. user-specific file*
*#  3. system-wide file*
*# Any configuration value is only changed the first time it is set.*
*# Thus, host-specific definitions should be at the beginning of the*
*# configuration file, and defaults at the end.*

*# Site-wide defaults for some commonly used options.  For a comprehensive*
*# list of available options, their meanings and defaults, please see the*
*# ssh_config(5) man page.*

Host *
*#   ForwardAgent no*
*#   ForwardX11 no*
*#   ForwardX11Trusted yes*
*#   PasswordAuthentication yes*
*#   HostbasedAuthentication no*
*#   GSSAPIAuthentication no*
*#   GSSAPIDelegateCredentials no*
*#   GSSAPIKeyExchange no*
*#   GSSAPITrustDNS no*
*#   BatchMode no*
*#   CheckHostIP yes*
*#   AddressFamily any*
*#   ConnectTimeout 0*
*#   StrictHostKeyChecking ask*
*#   IdentityFile ~/.ssh/id_rsa*
*#   IdentityFile ~/.ssh/id_dsa*
*#   IdentityFile ~/.ssh/id_ecdsa*
*#   IdentityFile ~/.ssh/id_ed25519*
*#   Port 22*
*#   Protocol 2*
*#   Ciphers aes128-ctr,aes192-ctr,aes256-ctr,aes128-cbc,3des-cbc*
*#   MACs hmac-md5,hmac-sha1,umac-64@openssh.com*
*#   EscapeChar ~*
*#   Tunnel no*
*#   TunnelDevice any:any*
*#   PermitLocalCommand no*
*#   VisualHostKey no*
*#   ProxyCommand ssh -q -W %h:%p gateway.example.com*
*#   RekeyLimit 1G 1h*
    SendEnv LANG LC_*
    HashKnownHosts yes
    GSSAPIAuthentication yes

Host FQDN
  PasswordAuthentication no
  IdentityFile ~/.ssh/id_rsa
  CheckHostIP yes

(How come you knew how to format output in the original post, but didn’t do it this time?)

I think something is fishy around keys, from the recent output:

debug1: Offering public key: RSA SHA256:GCOmXVN7463SEXRS+2MqMPh9l6X/+2J1ZvGHCm/6aiQ /home/user/.ssh/id_rsa

but that is not the fingerprint in the error message you quoted in the first post:

debug1: Will attempt key: /home/user/.ssh/id_rsa RSA SHA256:VX3JDVrZYNtFpFUhiQR11IYdRCotA/yl/H0DodwKqRY agent

Could it be that you’re using two different users, but just change both usernames to “user” before posting here? Other than that, I can’t imagine what you can have done to make that happen.

I’m using the same key pair. Also tried looking for an auth log, or something like secure log on the gitlab server but haven’t found one yet.

Hi, I tried attaching the ip address directly again to my gitlab server, tried git via ssh on FQDN and IP_address and it worked without error.

image_2022-12-10_101817554

But when I revert back to my initial planned setup where gitlab server is proxied on an apache server, I got the error permission denied again.

image_2022-12-10_102058422

I would guess apache doesn’t relay ssh traffic properly in a setup like that.

Correct. Apache can be used as a reverse HTTP proxy but does not understand nor forward other protocols such as SSH. Another option would be to use a reverse SSH tunnel to connect to the FQDN server, and then to the gitlab_server host, something like this in a terminal: ssh git@FQDN -L 9000:gitlab_server:22 and on a second terminal, ssh git@127.0.0.1 -p 9000 (untested).

Not sure if that works reliably with Git, though; never done that. I’d suggest using HTTPS as a Git protocol here or providing the GitLab Server with its own dedicated VM and removing the other VHosts that could influence performance.

Thank you @grove and @dnsmichi. Noted on the issue on setting up the gitlab like this.