Enabled SSH in gitlab

How to Use GitLab Self-managed
1

i have a gitlab omnibus setup for atleast 65 users and 155 repositories

i want to enable SSH for all my users. i tried enabling it by adding the neccessary configurations for port 22 in my NLB

As NLB creates an IP per AZ, mine is ap-southeast-2a and 2c, at this moment my SSH fails as it fails the IP Check as it hits on different server each time.

i need to enable it for everyone without adding personal IPs of everyone in the Security Groups.

what else can i do?

2

what do you get if you run this command from the terminal

ssh -vvv git@your-domain.com

Note: replace your-domain.com with your real domain.

3

t2b@t2b-Latitude-3420:~/.ssh$ ssh -vvv gitlab.thoughtstobinarycom
OpenSSH_8.2p1 Ubuntu-4ubuntu0.12, OpenSSL 1.1.1f 31 Mar 2020
debug1: Reading configuration data /home/t2b/.ssh/config
debug1: /home/t2b/.ssh/config line 5: Applying options for gitlab.thoughtstobinarycom
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug2: resolving “gitlab.thoughtstobinarycom” port 22
debug2: ssh_connect_direct
debug1: Connecting to gitlab.thoughtstobinarycom [52.63.46.125] port 22.
debug1: Connection established.
debug1: identity file /home/t2b/.ssh/codecommit type 0
debug1: identity file /home/t2b/.ssh/codecommit-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.12
debug1: Remote protocol version 2.0, remote software version OpenSSH_8.7
debug1: match: OpenSSH_8.7 pat OpenSSH* compat 0x04000000
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to gitlab.thoughtstobinarycom:22 as ‘git’
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libsshorg,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-g
roup-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,ext-info-c,kex-strict-c-v00@opens

debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01@opensshcom,ecdsa-sha2-nistp384-cert-v01@opensshcom,ecdsa-sha2-nistp521-cert-v01@open
sshcom,sk-ecdsa-sha2-nistp256-cert-v01@opensshcom,ssh-ed25519-cert-v01@opensshcom,sk-ssh-ed25519-cert-v01@openssh.com,rsa-sha2-512-cert-v01@o
pensshcom,rsa-sha2-256-cert-v01@opensshcom,ssh-rsa-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ecdsa-s
ha2-nistp256@opensshcom,ssh-ed25519,sk-ssh-ed25519@openssh.com,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug2: ciphers ctos: chacha20-poly1305@opensshcom,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@opensshcom
debug2: ciphers stoc: chacha20-poly1305@opensshcom,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@opensshcom
debug2: MACs ctos: umac-64-etm@opensshcom,umac-128-etm@openssh.com,hmac-sha2-256-etm@opensshcom,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@op
ensshcom,umac-64@opensshcom,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@opensshcom,umac-128-etm@openssh.com,hmac-sha2-256-etm@opensshcom,hmac-sha2-512-etm@opensshcom,hmac-sha1-etm@op
ensshcom,umac-64@opensshcom,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@opensshcom,zlib
debug2: compression stoc: none,zlib@opensshcom,zlib
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libsshorg,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-g
roup-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,kex-strict-s-v00@opensshcom
debug2: host key algorithms: ecdsa-sha2-nistp256,ssh-ed25519
debug2: ciphers ctos: aes256-gcm@opensshcom,chacha20-poly1305@opensshcom,aes256-ctr,aes128-gcm@opensshcom,aes128-ctr
debug2: ciphers stoc: aes256-gcm@opensshcom,chacha20-poly1305@opensshcom,aes256-ctr,aes128-gcm@opensshcom,aes128-ctr
debug2: MACs ctos: hmac-sha2-256-etm@opensshcom,hmac-sha1-etm@opensshcom,umac-128-etm@opensshcom,hmac-sha2-512-etm@opensshcom,hmac-sha2-256,
hmac-sha1,umac-128@opensshcom,hmac-sha2-512
debug2: MACs stoc: hmac-sha2-256-etm@opensshcom,hmac-sha1-etm@opensshcom,umac-128-etm@opensshcom,hmac-sha2-512-etm@opensshcom,hmac-sha2-256,
hmac-sha1,umac-128@opensshcom,hmac-sha2-512
debug2: compression ctos: none,zlib@opensshcom
debug2: compression stoc: none,zlib@openssh.com
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug3: will use strict KEX ordering
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305@opensshcom MAC: compression: none
debug1: kex: client->server cipher: chacha20-poly1305@opensshcom MAC: compression: none
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:zMELO1eDtp/o0z1YEBVysS137NyAbldWoI0EvCLdPO4
The authenticity of host ‘gitlab.thoughtstobinarycom (52.63.46.125)’ can’t be established.
ECDSA key fingerprint is SHA256:zMELO1eDtp/o0z1YEBVysS137NyAbldWoI0EvCLdPO4.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added ‘gitlab.thoughtstobinarycom,52.63.46.125’ (ECDSA) to the list of known hosts.
debug3: send packet: type 21
debug1: resetting send seqnr 3
debug2: set_newkeys: mode 1
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug1: resetting read seqnr 3
debug1: SSH2_MSG_NEWKEYS received
debug2: set_newkeys: mode 0
debug1: rekey in after 134217728 blocks
debug1: Will attempt key: /home/t2b/.ssh/codecommit RSA SHA256:tt+2XkOkfzvcC2rqlfHm199fs35sqew0rY8QFbS2h4A explicit agent
debug2: pubkey_prepare: done
debug3: send packet: type 5
debug3: receive packet: type 7
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,sk-ssh-ed25519@opensshcom,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp25
6,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ecdsa-sha2-nistp256@opensshcom,webauthn-sk-ecdsa-sha2-nistp256@opensshcom>
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic
debug3: start over, passed a different list publickey,gssapi-keyex,gssapi-with-mic
debug3: preferred gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup gssapi-with-mic
debug3: remaining preferred: publickey,keyboard-interactive,password
debug3: authmethod_is_enabled gssapi-with-mic
debug1: Next authentication method: gssapi-with-mic
debug1: Unspecified GSS failure. Minor code may provide more information
No Kerberos credentials available (default cache: FILE:/tmp/krb5cc_1001)

debug1: Unspecified GSS failure. Minor code may provide more information
No Kerberos credentials available (default cache: FILE:/tmp/krb5cc_1001)

debug2: we did not send a packet, disable method
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering public key: /home/t2b/.ssh/codecommit RSA SHA256:tt+2XkOkfzvcC2rqlfHm199fs35sqew0rY8QFbS2h4A explicit agent
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 60
debug1: Server accepts key: /home/t2b/.ssh/codecommit RSA SHA256:tt+2XkOkfzvcC2rqlfHm199fs35sqew0rY8QFbS2h4A explicit agent
debug3: sign_and_send_pubkey: RSA SHA256:tt+2XkOkfzvcC2rqlfHm199fs35sqew0rY8QFbS2h4A
debug3: sign_and_send_pubkey: signing using rsa-sha2-512 SHA256:tt+2XkOkfzvcC2rqlfHm199fs35sqew0rY8QFbS2h4A
debug3: send packet: type 50
debug3: receive packet: type 52
debug1: Authentication succeeded (publickey).
Authenticated to gitlab.thoughtstobinarycom ([52.63.46.125]:22).
debug1: channel 0: new [client-session]
debug3: ssh_session2_open: channel_new: 0
debug2: channel 0: send open
debug3: send packet: type 90
debug1: Requesting no-more-sessions@opensshcom
debug3: send packet: type 80
debug1: Entering interactive session.
debug1: pledge: network
debug3: receive packet: type 80
debug1: client_input_global_request: rtype hostkeys-00@opensshcom want_reply 0
debug3: receive packet: type 4
debug1: Remote: /var/opt/gitlab/.ssh/authorized_keys:6: key options: command user-rc
debug3: receive packet: type 4
debug1: Remote: /var/opt/gitlab/.ssh/authorized_keys:6: key options: command user-rc
debug3: receive packet: type 91
debug2: channel_input_open_confirmation: channel 0: callback start
debug2: fd 3 setting TCP_NODELAY
debug3: ssh_packet_set_tos: set IP_TOS 0x10
debug2: client_session2_setup: id 0
debug2: channel 0: request pty-req confirm 1
debug3: send packet: type 98
debug1: Sending environment.
debug3: Ignored env SHELL
debug3: Ignored env SESSION_MANAGER
debug3: Ignored env WINDOWID
debug3: Ignored env QT_ACCESSIBILITY
debug3: Ignored env COLORTERM
debug3: Ignored env XDG_CONFIG_DIRS
debug3: Ignored env NVM_INC
debug3: Ignored env HISTCONTROL
debug3: Ignored env XDG_MENU_PREFIX
debug3: Ignored env GNOME_DESKTOP_SESSION_ID
debug3: Ignored env rvm_prefix
debug3: Ignored env HISTSIZE
debug3: Ignored env LANGUAGE
debug3: Ignored env GNOME_SHELL_SESSION_MODE
debug3: Ignored env SSH_AUTH_SOCK
debug3: Ignored env HISTTIMEFORMAT
debug3: Ignored env SHELL_SESSION_ID
debug3: Ignored env MY_RUBY_HOME
debug3: Ignored env XMODIFIERS
debug3: Ignored env DESKTOP_SESSION
debug3: Ignored env SSH_AGENT_PID
debug3: Ignored env GTK_MODULES
debug3: Ignored env RUBY_VERSION
debug3: Ignored env PWD
debug3: Ignored env XDG_SESSION_DESKTOP
debug3: Ignored env LOGNAME
debug3: Ignored env XDG_SESSION_TYPE
debug3: Ignored env rvm_version
debug3: Ignored env GPG_AGENT_INFO
debug3: Ignored env XAUTHORITY
debug3: Ignored env GJS_DEBUG_TOPICS
debug3: Ignored env WINDOWPATH
debug3: Ignored env HOME
debug3: Ignored env USERNAME
debug3: Ignored env IM_CONFIG_PHASE
debug1: Sending env LANG = en_IN
debug2: channel 0: request env confirm 0
debug3: send packet: type 98
debug3: Ignored env LS_COLORS
debug3: Ignored env XDG_CURRENT_DESKTOP
debug3: Ignored env KONSOLE_DBUS_SERVICE
debug3: Ignored env KONSOLE_DBUS_SESSION
debug3: Ignored env PROFILEHOME
debug3: Ignored env INVOCATION_ID
debug3: Ignored env KONSOLE_VERSION
debug3: Ignored env MANAGERPID
debug3: Ignored env GJS_DEBUG_OUTPUT
debug3: Ignored env NVM_DIR
debug3: Ignored env rvm_bin_path
debug3: Ignored env GEM_PATH
debug3: Ignored env GEM_HOME
debug3: Ignored env LESSCLOSE
debug3: Ignored env XDG_SESSION_CLASS
debug3: Ignored env TERM
debug3: Ignored env LESSOPEN
debug3: Ignored env USER
debug3: Ignored env COLORFGBG
debug3: Ignored env DISPLAY
debug3: Ignored env SHLVL
debug3: Ignored env NVM_CD_FLAGS
debug3: Ignored env QT_IM_MODULE
debug3: Ignored env XDG_RUNTIME_DIR
debug3: Ignored env JOURNAL_STREAM
debug3: Ignored env XDG_DATA_DIRS
debug3: Ignored env PATH
debug3: Ignored env GDMSESSION
debug3: Ignored env HISTFILESIZE
debug3: Ignored env DBUS_SESSION_BUS_ADDRESS
debug3: Ignored env NVM_BIN
debug3: Ignored env IRBRC
debug3: Ignored env GIO_LAUNCHED_DESKTOP_FILE_PID
debug3: Ignored env GIO_LAUNCHED_DESKTOP_FILE
debug3: Ignored env rvm_path
debug3: Ignored env KONSOLE_DBUS_WINDOW
debug3: Ignored env OLDPWD
debug3: Ignored env _
debug2: channel 0: request shell confirm 1
debug3: send packet: type 98
debug2: channel_input_open_confirmation: channel 0: callback done
debug2: channel 0: open confirm rwindow 0 rmax 32768
debug3: receive packet: type 100
debug2: channel_input_status_confirm: type 100 id 0
PTY allocation request failed on channel 0
debug2: channel 0: rcvd adjust 2097152
debug3: receive packet: type 99
debug2: channel_input_status_confirm: type 99 id 0
debug2: shell request accepted on channel 0
Welcome to GitLab, @navya.sharma!
debug3: receive packet: type 96
debug2: channel 0: rcvd eof
debug2: channel 0: output open → drain
debug2: channel 0: obuf empty
debug2: channel 0: chan_shutdown_write (i0 o1 sock -1 wfd 5 efd 6 [write])
debug2: channel 0: output drain → closed
debug3: receive packet: type 98
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug3: receive packet: type 98
debug1: client_input_channel_req: channel 0 rtype eow@opensshcom reply 0
debug2: channel 0: rcvd eow
debug2: channel 0: chan_shutdown_read (i0 o3 sock -1 wfd 4 efd 6 [write])
debug2: channel 0: input open → closed
debug3: receive packet: type 97
debug2: channel 0: rcvd close
debug3: channel 0: will not send data after close
debug2: channel 0: almost dead
debug2: channel 0: gc: notify user
debug2: channel 0: gc: user detached
debug2: channel 0: send close
debug3: send packet: type 97
debug2: channel 0: is dead
debug2: channel 0: garbage collecting
debug1: channel 0: free: client-session, nchannels 1
debug3: channel 0: status: The following connections are open:
#0 client-session (t4 r0 i3/0 o3/0 e[write]/0 fd -1/-1/6 sock -1 cc -1)

debug3: send packet: type 1
debug3: fd 1 is not O_NONBLOCK
Connection to gitlab.thoughtstobinarycom closed.
Transferred: sent 3964, received 2788 bytes, in 0.7 seconds
Bytes per second: sent 5687.3, received 4000.0
debug1: Exit status 0
t2b@t2b-Latitude-3420:~/.ssh$ ssh -vvv gitlab.thoughtstobinarycom
OpenSSH_8.2p1 Ubuntu-4ubuntu0.12, OpenSSL 1.1.1f 31 Mar 2020
debug1: Reading configuration data /home/t2b/.ssh/config
debug1: /home/t2b/.ssh/config line 5: Applying options for gitlab.thoughtstobinarycom
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug2: resolving “gitlab.thoughtstobinarycom” port 22
debug2: ssh_connect_direct
debug1: Connecting to gitlab.thoughtstobinarycom [54.66.72.25] port 22.
debug1: Connection established.
debug1: identity file /home/t2b/.ssh/codecommit type 0
debug1: identity file /home/t2b/.ssh/codecommit-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.12
debug1: Remote protocol version 2.0, remote software version OpenSSH_8.7
debug1: match: OpenSSH_8.7 pat OpenSSH* compat 0x04000000
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to gitlab.thoughtstobinarycom:22 as ‘git’
debug3: hostkeys_foreach: reading file “/home/t2b/.ssh/known_hosts”
debug3: record_hostkey: found key type ECDSA in file /home/t2b/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys from gitlab.thoughtstobinarycom
debug3: order_hostkeyalgs: have matching best-preference key type ecdsa-sha2-nistp256-cert-v01@opensshcom, using HostkeyAlgorithms verbatim
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libsshorg,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-g
roup-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,ext-info-c,kex-strict-c-v00@opens

debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01@openssh..com,ecdsa-sha2-nistp384-cert-v01@openssh.\com,ecdsa-sha2-nistp521-cert-v01@open
ssh./com,sk-ecdsa-sha2-nistp256-cert-v01@openssh./com,ssh-ed25519-cert-v01@openssh..com,sk-ssh-ed25519-cert-v01@openssh..com,rsa-sha2-512-cert-v01@o
penssh./com,rsa-sha2-256-cert-v01@openssh./com,ssh-rsa-cert-v01@openssh./com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ecdsa-s
ha2-nistp256@openssh./com,ssh-ed25519,sk-ssh-ed25519@openssh./com,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug2: ciphers ctos: chacha20-poly1305@opensshcom,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@opensshcom
debug2: ciphers stoc: chacha20-poly1305@opensshcom,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@opensshcom
debug2: MACs ctos: umac-64-etm@opensshcom,umac-128-etm@openssh.com,hmac-sha2-256-etm@opensshcom,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@op
ensshcom,umac-64@opensshcom,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@opensshcom,umac-128-etm@openssh.com,hmac-sha2-256-etm@opensshcom,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@op
ensshcom,umac-64@opensshcom,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@opensshcom,zlib
debug2: compression stoc: none,zlib@opensshcom,zlib
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libsshorg,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-g
roup-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,kex-strict-s-v00@opensshcom
debug2: host key algorithms: ecdsa-sha2-nistp256,ssh-ed25519
debug2: ciphers ctos: aes256-gcm@opensshcom,chacha20-poly1305@opensshcom,aes256-ctr,aes128-gcm@opensshcom,aes128-ctr
debug2: ciphers stoc: aes256-gcm@opensshcom,chacha20-poly1305@opensshcom,aes256-ctr,aes128-gcm@opensshcom,aes128-ctr
debug2: MACs ctos: hmac-sha2-256-etm@opensshcom,hmac-sha1-etm@opensshcom,umac-128-etm@opensshcom,hmac-sha2-512-etm@opensshcom,hmac-sha2-256,
hmac-sha1,umac-128@opensshcom,hmac-sha2-512
debug2: MACs stoc: hmac-sha2-256-etm@opensshcom,hmac-sha1-etm@opensshcom,umac-128-etm@opensshcom,hmac-sha2-512-etm@opensshcom,hmac-sha2-256,
hmac-sha1,umac-128@opensshcom,hmac-sha2-512
debug2: compression ctos: none,zlib@opensshcom
debug2: compression stoc: none,zlib@opensshcom
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug3: will use strict KEX ordering
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305@opensshcom MAC: compression: none
debug1: kex: client->server cipher: chacha20-poly1305@opensshcom MAC: compression: none
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:Stn67FTtlwwA0KVAzxYh7f1C2x4D3pQCdBCHu2uZPJo
debug3: hostkeys_foreach: reading file “/home/t2b/.ssh/known_hosts”
debug3: record_hostkey: found key type ECDSA in file /home/t2b/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys from gitlab.thoughtstobinarycom
debug3: hostkeys_foreach: reading file “/home/t2b/.ssh/known_hosts”
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: POSSIBLE DNS SPOOFING DETECTED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
The ECDSA host key for gitlab.thoughtstobinarycom has changed,
and the key for the corresponding IP address 54.66.72.25
is unknown. This could either mean that
DNS SPOOFING is happening or the IP address for the host
and its host key have changed at the same time.
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ECDSA key sent by the remote host is
SHA256:Stn67FTtlwwA0KVAzxYh7f1C2x4D3pQCdBCHu2uZPJo.
Please contact your system administrator.
Add correct host key in /home/t2b/.ssh/known_hosts to get rid of this message.
Offending ECDSA key in /home/t2b/.ssh/known_hosts:1
remove with:
ssh-keygen -f “/home/t2b/.ssh/known_hosts” -R “gitlab.thoughtstobinarycom”
ECDSA host key for gitlab.thoughtstobinarycom has changed and you have requested strict checking.
Host key verification failed.
t2b@t2b-Latitude-3420:~/.ssh$

4

Do you have something like an IP or load balancer? When I ask for your DNS information of your domain I got this two IPs registered for the domain. Maybe it is doing your ssh clients are going from one ip to the other ip.

dig gitlab.thoughtstobinary.com

; <<>> DiG 9.10.6 <<>> gitlab.thoughtstobinary.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 7037
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1220
;; QUESTION SECTION:
;gitlab.thoughtstobinary.com. IN A

;; ANSWER SECTION:
gitlab.thoughtstobinary.com. 56 IN A 52.63.46.125
gitlab.thoughtstobinary.com. 56 IN A 54.66.72.25

;; Query time: 52 msec
;; SERVER: 190.184.56.72#53(190.184.56.72)
;; WHEN: Tue Sep 09 11:32:28 CST 2025
;; MSG SIZE rcvd: 88

5

He didn’t follow your commands correctly. In his debug output:

You can still see that he didn’t use git@gitlab.thoughtstobinary.com which means he didn’t connect as git user, but as t2b user which won’t work. So it could never find the SSH key he uploaded to gitlab.

@navya when using gitlab, the ssh connection user is always git followed by your servername.

Also, the NLB should have a port forwarding the SSH connection to the Gitlab server, port 22 cannot be used since it’s most likely already in use by the NLB. That would also confirm the fact the SSH host key changes each time he tries to connect to the host he then gets one key from one NLM and later another key from the other NLB. If it was a single Gitlab server behind the NLB, with the correct port forwarded then the SSH host key would always be the same. Which suggests the NLB’s are not configured correctly as well. For example, port 22222 on the NLB being forwarded to port 22 on the Gitlab server, then using:

ssh -p 22222 git@gitlab.thoughtstobinary.com
6

You are absolutely right as always @iwalker about he didn’t run the command correctly. Just for fun, I ran the command on my computer and I get this output.. in line 10 we can see that the connection is stablished to the port 22. Obviously, I can’t authenticate.

ssh -vvv git@gitlab.thoughtstobinary.com
OpenSSH_9.9p2, LibreSSL 3.3.6
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 21: include /etc/ssh/ssh_config.d/* matched no files
debug1: /etc/ssh/ssh_config line 54: Applying options for *
debug3: expanded UserKnownHostsFile ‘~/.ssh/known_hosts’ → ‘/Users/brivas/.ssh/known_hosts’
debug3: expanded UserKnownHostsFile ‘~/.ssh/known_hosts2’ → ‘/Users/brivas/.ssh/known_hosts2’
debug1: Authenticator provider $SSH_SK_PROVIDER did not resolve; disabling
debug3: channel_clear_timeouts: clearing
debug1: Connecting to gitlab.thoughtstobinary.com port 22.
debug1: Connection established.
debug1: identity file /Users/brivas/.ssh/id_rsa type -1
debug1: identity file /Users/brivas/.ssh/id_rsa-cert type -1
debug1: identity file /Users/brivas/.ssh/id_ecdsa type -1
debug1: identity file /Users/brivas/.ssh/id_ecdsa-cert type -1
debug1: identity file /Users/brivas/.ssh/id_ecdsa_sk type -1
debug1: identity file /Users/brivas/.ssh/id_ecdsa_sk-cert type -1
debug1: identity file /Users/brivas/.ssh/id_ed25519 type 3
debug1: identity file /Users/brivas/.ssh/id_ed25519-cert type -1
debug1: identity file /Users/brivas/.ssh/id_ed25519_sk type -1
debug1: identity file /Users/brivas/.ssh/id_ed25519_sk-cert type -1
debug1: identity file /Users/brivas/.ssh/id_xmss type -1
debug1: identity file /Users/brivas/.ssh/id_xmss-cert type -1
debug1: identity file /Users/brivas/.ssh/id_dsa type -1
debug1: identity file /Users/brivas/.ssh/id_dsa-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_9.9
debug1: Remote protocol version 2.0, remote software version OpenSSH_8.7
debug1: compat_banner: match: OpenSSH_8.7 pat OpenSSH* compat 0x04000000
debug3: fd 6 is O_NONBLOCK
debug1: Authenticating to gitlab.thoughtstobinary.com:22 as ‘git’
debug3: record_hostkey: found key type ED25519 in file /Users/brivas/.ssh/known_hosts:11
debug3: load_hostkeys_file: loaded 1 keys from gitlab.thoughtstobinary.com
debug1: load_hostkeys: fopen /Users/brivas/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug3: order_hostkeyalgs: have matching best-preference key type ssh-ed25519-cert-v01@openssh.com, using HostkeyAlgorithms verbatim
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: sntrup761x25519-sha512,sntrup761x25519-sha512@openssh.com,mlkem768x25519-sha256,curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,ext-info-c,kex-strict-c-v00@openssh.com
debug2: host key algorithms: ssh-ed25519-cert-v01@openssh.com,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,sk-ssh-ed25519-cert-v01@openssh.com,sk-ecdsa-sha2-nistp256-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ssh-ed25519@openssh.com,sk-ecdsa-sha2-nistp256@openssh.com,rsa-sha2-512,rsa-sha2-256
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com
debug2: compression stoc: none,zlib@openssh.com
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,kex-strict-s-v00@openssh.com
debug2: host key algorithms: ecdsa-sha2-nistp256,ssh-ed25519
debug2: ciphers ctos: aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,aes256-ctr,aes128-gcm@openssh.com,aes128-ctr
debug2: ciphers stoc: aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,aes256-ctr,aes128-gcm@openssh.com,aes128-ctr
debug2: MACs ctos: hmac-sha2-256-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha2-256,hmac-sha1,umac-128@openssh.com,hmac-sha2-512
debug2: MACs stoc: hmac-sha2-256-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha2-256,hmac-sha1,umac-128@openssh.com,hmac-sha2-512
debug2: compression ctos: none,zlib@openssh.com
debug2: compression stoc: none,zlib@openssh.com
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug3: kex_choose_conf: will use strict KEX ordering
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ssh-ed25519
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: compression: none
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: SSH2_MSG_KEX_ECDH_REPLY received
debug1: Server host key: ssh-ed25519 SHA256:XeDr8C+AULOW6cTKWTlDvBi0D+XpG/kAwbS5I4tGkNw
debug3: record_hostkey: found key type ED25519 in file /Users/brivas/.ssh/known_hosts:11
debug3: load_hostkeys_file: loaded 1 keys from gitlab.thoughtstobinary.com
debug1: load_hostkeys: fopen /Users/brivas/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: Host ‘gitlab.thoughtstobinary.com’ is known and matches the ED25519 host key.
debug1: Found key in /Users/brivas/.ssh/known_hosts:11
debug3: send packet: type 21
debug1: ssh_packet_send2_wrapped: resetting send seqnr 3
debug2: ssh_set_newkeys: mode 1
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug1: ssh_packet_read_poll2: resetting read seqnr 3
debug1: SSH2_MSG_NEWKEYS received
debug2: ssh_set_newkeys: mode 0
debug1: rekey in after 134217728 blocks
debug2: KEX algorithms: sntrup761x25519-sha512,sntrup761x25519-sha512@openssh.com,mlkem768x25519-sha256,curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,ext-info-c,kex-strict-c-v00@openssh.com
debug2: host key algorithms: ssh-ed25519-cert-v01@openssh.com,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,sk-ssh-ed25519-cert-v01@openssh.com,sk-ecdsa-sha2-nistp256-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ssh-ed25519@openssh.com,sk-ecdsa-sha2-nistp256@openssh.com,rsa-sha2-512,rsa-sha2-256
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com
debug2: compression stoc: none,zlib@openssh.com
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug3: send packet: type 5
debug3: receive packet: type 7
debug1: SSH2_MSG_EXT_INFO received
debug3: kex_input_ext_info: extension server-sig-algs
debug1: kex_ext_info_client_parse: server-sig-algs=<ssh-ed25519,sk-ssh-ed25519@openssh.com,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ecdsa-sha2-nistp256@openssh.com,webauthn-sk-ecdsa-sha2-nistp256@openssh.com>
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic
debug3: start over, passed a different list publickey,gssapi-keyex,gssapi-with-mic
debug3: preferred publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug3: ssh_get_authentication_socket_path: path ‘/private/tmp/com.apple.launchd.Hy4BwzdTa0/Listeners’
debug1: get_agent_identities: bound agent to hostkey
debug1: get_agent_identities: ssh_fetch_identitylist: agent contains no identities
debug1: Will attempt key: /Users/brivas/.ssh/id_rsa
debug1: Will attempt key: /Users/brivas/.ssh/id_ecdsa
debug1: Will attempt key: /Users/brivas/.ssh/id_ecdsa_sk
debug1: Will attempt key: /Users/brivas/.ssh/id_ed25519 ED25519 SHA256:7oK/zLoduknZjSmHVVMf0ov73Gdi1EQdoX/8fLcW+mQ
debug1: Will attempt key: /Users/brivas/.ssh/id_ed25519_sk
debug1: Will attempt key: /Users/brivas/.ssh/id_xmss
debug1: Will attempt key: /Users/brivas/.ssh/id_dsa
debug2: pubkey_prepare: done
debug1: Trying private key: /Users/brivas/.ssh/id_rsa
debug3: no such identity: /Users/brivas/.ssh/id_rsa: No such file or directory
debug1: Trying private key: /Users/brivas/.ssh/id_ecdsa
debug3: no such identity: /Users/brivas/.ssh/id_ecdsa: No such file or directory
debug1: Trying private key: /Users/brivas/.ssh/id_ecdsa_sk
debug3: no such identity: /Users/brivas/.ssh/id_ecdsa_sk: No such file or directory
debug1: Offering public key: /Users/brivas/.ssh/id_ed25519 ED25519 SHA256:7oK/zLoduknZjSmHVVMf0ov73Gdi1EQdoX/8fLcW+mQ
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic
debug1: Trying private key: /Users/brivas/.ssh/id_ed25519_sk
debug3: no such identity: /Users/brivas/.ssh/id_ed25519_sk: No such file or directory
debug1: Trying private key: /Users/brivas/.ssh/id_xmss
debug3: no such identity: /Users/brivas/.ssh/id_xmss: No such file or directory
debug1: Trying private key: /Users/brivas/.ssh/id_dsa
debug3: no such identity: /Users/brivas/.ssh/id_dsa: No such file or directory
debug2: we did not send a packet, disable method
debug1: No more authentication methods to try.
git@gitlab.thoughtstobinary.com: Permission denied (publickey,gssapi-keyex,gssapi-with-mic).

1 Like
7

@bayardo.rivas yep, at this point yours doesn’t work because the SSH key isn’t uploaded to his gitlab. But assuming you repeat it multiple times and get one of the other two IP addresses that the DNS record gitlab.thoughtstobinary.com has, then provided that the SSH key doesn’t change, it would at least confirm whether the NLB’s are configured correctly or not. Because if the NLB’s also have their own SSH port, then the LB itself won’t be able to listen on port 22 as well because it would already exist. And if the SSH key changes, that means it’s not configured correctly.

From what we know so far, or at least can guess, there is only one Gitlab server behind the NLB’s, so it’s not possible for the SSH host key to change unless it’s connecting to a totally different SSH instance. And if it is that would suggest a Gitlab cluster behind the NLB’s which also isn’t theoretically configured correctly.

I personally prefer to restrict SSH access for administration tasks only, and thus I only use https for push, pull or whatever else I do on my Gitlab servers. But obviously some people like to use that, which is fine assuming it’s configured correctly in the first place :slight_smile:

8

yes im using a Network load balancer and i have two gitlab instances in 2 different azs and hence the multiple IPS as an NLB creates a static IP per az @bayardo.rivas @iwalker

9

archnasharma@archnas-MacBook-Air .ssh % ssh -T git@gitlab.thoughtstobinary

Welcome to GitLab, @navya.sharma
archnasharma@archnas-MacBook-Air .ssh % ssh -T git@gitlab.thoughtstobinary

git@gitlab.thoughtstobinary: Permission denied (publickey,gssapi-keyex,gssapi-with-mic).
archnasharma@archnas-MacBook-Air .ssh % ssh -T git@gitlab.thoughtstobinary

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: POSSIBLE DNS SPOOFING DETECTED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
The ECDSA host key for gitlab.thoughtstobinary has changed,
and the key for the corresponding IP address 54.66.72.25
is unknown. This could either mean that
DNS SPOOFING is happening or the IP address for the host
and its host key have changed at the same time.
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ECDSA key sent by the remote host is
SHA256:.
Please contact your system administrator.
Add correct host key in /Users/archnasharma/.ssh/known_hosts to get rid of this message.
Offending ECDSA key in /Users/archnasharma/.ssh/known_hosts:4
ECDSA host key for gitlab.thoughtstobinary has changed and you have requested strict checking.
Host key verification failed.
archnasharma@archnas-MacBook-Air .ssh %

i tried this with git@gitlab and before as well the user was set as git in my config @bayardo.rivas @iwalker

10

Are the two Gitlab instances mirrored? The same data is on both instances? I think this is configured wrong. The only way it will work this way is by ignoring the host SSH key because you are connecting to two different servers using the same DNS entry. You can do that by using:

StrictHostKeyChecking no

but then SSH will not be as secure as it’s meant to be.

I would suggest reading the Gitlab documentation here: Reference architectures | GitLab Docs on how to configure correctly.

11

yes we have two gitlab instances for high avaliability and we scale them as required