Remote: Failed to get username: Internal API unreachable

CentOS 7 and Gitlab-CE + auth via LDAP

GitLab 13.5.2 (187cae1b32b)
GitLab Shell13.11.0
GitLab Workhorsev8.51.0
GitLab APIv4
Ruby2.6.6p146
Rails6.0.3.3
PostgreSQL11.9

I have gitlab in use for quite a while now and I’ve updated it already a couple of times, but right now it seems the update has broken something.
I thought maybe an issue with the old RSA key, so I’ve created a new ED25519 and pasted the public key inside my gitlabs profile.
When I execute git push I get:

remote: 
remote: ========================================================================
remote: 
remote: Internal API unreachable
remote: 
remote: ========================================================================
remote: 
fatal: Could not read from remote repository.

So I’ve tried the following:

ssh -vT git@gitlab.mydomain.com
OpenSSH_8.1p1, LibreSSL 2.7.3
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 47: Applying options for *
debug1: Connecting to gitlab.mydomain.com port 22.
debug1: Connection established.
debug1: identity file /Users/MYUSERNAME/.ssh/id_rsa type 0
debug1: identity file /Users/MYUSERNAME/.ssh/id_rsa-cert type -1
debug1: identity file /Users/MYUSERNAME/.ssh/id_dsa type -1
debug1: identity file /Users/MYUSERNAME/.ssh/id_dsa-cert type -1
debug1: identity file /Users/MYUSERNAME/.ssh/id_ecdsa type -1
debug1: identity file /Users/MYUSERNAME/.ssh/id_ecdsa-cert type -1
debug1: identity file /Users/MYUSERNAME/.ssh/id_ed25519 type 3
debug1: identity file /Users/MYUSERNAME/.ssh/id_ed25519-cert type -1
debug1: identity file /Users/MYUSERNAME/.ssh/id_xmss type -1
debug1: identity file /Users/MYUSERNAME/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.1
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.4
debug1: match: OpenSSH_7.4 pat OpenSSH_7.0*,OpenSSH_7.1*,OpenSSH_7.2*,OpenSSH_7.3*,OpenSSH_7.4*,OpenSSH_7.5*,OpenSSH_7.6*,OpenSSH_7.7* compat 0x04000002
debug1: Authenticating to gitlab.mydomain.com:22 as 'git'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:o/OOAEPEgFNyzmx1hSwV5B12L70WMCjEPAPV5sNnQkI
debug1: Host 'gitlab.mydomain.com' is known and matches the ECDSA host key.
debug1: Found key in /Users/MYUSERNAME/.ssh/known_hosts:19
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey in after 134217728 blocks
debug1: Will attempt key: /Users/MYUSERNAME/.ssh/id_rsa RSA SHA256:123456789etc.
debug1: Will attempt key: /Users/MYUSERNAME/.ssh/id_dsa 
debug1: Will attempt key: /Users/MYUSERNAME/.ssh/id_ecdsa 
debug1: Will attempt key: /Users/MYUSERNAME/.ssh/id_ed25519 ED25519 SHA256:123456789etc.
debug1: Will attempt key: /Users/MYUSERNAME/.ssh/id_xmss 
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<rsa-sha2-256,rsa-sha2-512>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic
debug1: Next authentication method: publickey
debug1: Offering public key: /Users/MYUSERNAME/.ssh/id_rsa RSA SHA256:123456789etc.
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic
debug1: Trying private key: /Users/MYUSERNAME/.ssh/id_dsa
debug1: Trying private key: /Users/MYUSERNAME/.ssh/id_ecdsa
debug1: Offering public key: /Users/MYUSERNAME/.ssh/id_ed25519 ED25519 SHA256:123456789etc.
debug1: Server accepts key: /Users/MYUSERNAME/.ssh/id_ed25519 ED25519 SHA256:123456789etc.
debug1: Authentication succeeded (publickey).
Authenticated to gitlab.mydomain.com ([MYIPADDRESS]:22).
debug1: channel 0: new [client-session]
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.
debug1: pledge: network
debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0
debug1: Remote: Forced command.
debug1: Remote: Port forwarding disabled.
debug1: Remote: X11 forwarding disabled.
debug1: Remote: Agent forwarding disabled.
debug1: Remote: PTY allocation disabled.
debug1: Remote: Forced command.
debug1: Remote: Port forwarding disabled.
debug1: Remote: X11 forwarding disabled.
debug1: Remote: Agent forwarding disabled.
debug1: Remote: PTY allocation disabled.
debug1: Sending environment.
debug1: Sending env LANG = en_US.UTF-8
debug1: Sending env LC_CTYPE = UTF-8
remote: 
remote: ========================================================================
remote: 
remote: Failed to get username: Internal API unreachable
remote: 
remote: ========================================================================
remote: 
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug1: client_input_channel_req: channel 0 rtype eow@openssh.com reply 0
debug1: channel 0: free: client-session, nchannels 1
Transferred: sent 2480, received 3260 bytes, in 0.2 seconds
Bytes per second: sent 15291.7, received 20101.1
debug1: Exit status 1

Also done a gitlab check:

# gitlab-rake gitlab:check
Checking GitLab subtasks ...

Checking GitLab Shell ...

GitLab Shell: ... GitLab Shell version >= 13.11.0 ? ... OK (13.11.0)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Internal API available: OK
Redis available via internal API: OK
gitlab-shell self-check successful

Checking GitLab Shell ... Finished

Checking Gitaly ...

Gitaly: ... default ... OK

Checking Gitaly ... Finished

Checking Sidekiq ...

Sidekiq: ... Running? ... yes
Number of Sidekiq processes ... 1

Checking Sidekiq ... Finished

Checking Incoming Email ...

Incoming Email: ... Reply by email is disabled in config/gitlab.yml

Checking Incoming Email ... Finished

Checking LDAP ...

LDAP: ... Server: ldapmain
LDAP authentication... Success
LDAP users with access to your GitLab server (only showing the first 100 results)
    DN: cn=foo
    DN: cn=bar
    DN: cn=more

Checking LDAP ... Finished

Checking GitLab App ...

Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Projects have namespace: ... 
Foo / foo-foo ... yes
Foo / foo-bar ... yes
Foo / foo-more ... yes
whatever / General ... yes
A / B / C ... yes
Redis version >= 4.0.0? ... yes
Ruby version >= 2.5.3 ? ... yes (2.6.6)
Git version >= 2.24.0 ? ... yes (2.28.0)
Git user has default SSH configuration? ... yes
Active users: ... 2
Is authorized keys file accessible? ... yes
GitLab configured to store new projects in hashed storage? ... yes
All projects are in hashed storage? ... yes

Checking GitLab App ... Finished


Checking GitLab subtasks ... Finished

Am I missing something something, or is this simply a bug in the current release? It clearly tells me that debug1: Authentication succeeded (publickey). AND Authenticated to gitlab.mydomain.com ([MYIPADDRESS]:22).
So what could cause the issue?

Okay, found the culprit. Although I got a tail -f on selinux’s audit.log (it wasn’t showing any denials), but it blocked the action in the back.
After I temporarily swichted the modes (setenforce 0) it worked. I will have to investigate that a bit more, why it isn’t logging the denial and what the denial is.

After adding the following SELinux type enforcement rules, it finally worked:

#============= user_t ==============
allow user_t gitlab_shell_t:dir search;
allow user_t gitlab_shell_t:file { read open getattr };
allow user_t httpd_sys_content_t:dir search;

Unfortunately I haven’t found any other solution but to add this rules. In case anyone has a better solution, let me know.