Git ssh permission denied

Hey all,

I’ve deployed gitlab/gitlab-ce:9.5.2-ce.0 omnibus helm charts for kubernetes successfully and tried to clone my first empty project and am getting the following:

$ git clone git@gitlab.domain.com:user/hello.git
Cloning into 'hello'...
Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

I then tried ssh and am getting the following in verbose:

$ ssh -Tv git@domain.com
OpenSSH_7.4p1, LibreSSL 2.5.0
debug1: Reading configuration data /Users/user/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Connecting to gitlab.domain.com [172.0.0.2] port 22.
debug1: Connection established.
debug1: identity file /Users/user/.ssh/id_rsa type 1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/user/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/user/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/user/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/user/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/user/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/user/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/user/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.4
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.2p2 Ubuntu-4ubuntu2.2
debug1: match: OpenSSH_7.2p2 Ubuntu-4ubuntu2.2 pat OpenSSH* compat 0x04000000
debug1: Authenticating to gitlab.domain.com:22 as 'git'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256@libssh.org
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:5XCEWFXxCDbn3gxa2REa8//zuHsG4qwIKJuc8JNKxEw
debug1: Host 'domain.com' is known and matches the ECDSA host key.
debug1: Found key in /Users/user/.ssh/known_hosts:12
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<rsa-sha2-256,rsa-sha2-512>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /Users/user/.ssh/id_rsa
debug1: Authentications that can continue: publickey
debug1: Trying private key: /Users/user/.ssh/id_dsa
debug1: Trying private key: /Users/user/.ssh/id_ecdsa
debug1: Trying private key: /Users/user/.ssh/id_ed25519
debug1: No more authentication methods to try.
Permission denied (publickey).

I can see my public key added to authorized_users file after uploading in web ui. There is also an authorized_users.lock file that I read another user removed and it fixed the identical issue, but it did not affect any change here.

I’ve tried setting gitlab_shell['log_level'] = 'DEBUG' but no new info showed in the shell log file, so I can’t really tell where the problem is.

Thanks

after debugging with tcpdump, I found no packets were making it to the gitlab container on port 22. the router node had sshd running on all its ips so 22 traffic was terminating at the node. modified the hosts sshd config and now see packets making it to the gitlab container, but am still getting the same messages as before, albeit with gitlab container’s ssh host key so i can confirm i’m making it to the right endpoint.

and now finally the smoking gun from gitlab-ctl tail:

2017-10-06_22:19:51.41774 Authentication refused: bad ownership or modes for directory /gitlab-data
2017-10-06_22:19:51.42191 Connection closed by <IP_REDACTED> port 34720 [preauth]

relevant: