Gitlab-shell/bin/check Internal API available: FAILED - 401 Unauthorized

How to Use GitLab Upgrade
1

I’m in the process of upgrading my home gitlab installation after neglecting it for a year and a half and at the moment I’m having some fun with gitlab-shell.

This is running on gentoo using the gitlab overlay from

Given that I haven’t touched this installation for a long time it’s turning out to be somewhat complicated. I have successfully upgraded from 14.9.2 → 14.9.5 → 14.10.4 → 15.0.0 and everything seemed to be fine through these stages. The web ui was happy and bundle exec rake gitlab:check was all green. The latest upgrade I’ve done was from 15.0.0 → 15.0.5 and now bundle exec rake gitlab:check fails with the following

Checking GitLab Shell ...

GitLab Shell: ... GitLab Shell version >= 14.3.0 ? ... OK (14.13.0)
Running /opt/gitlab/gitlab-shell/bin/check
Internal API available: FAILED - 401 Unauthorized
gitlab-shell self-check failed
  Try fixing it:
  Make sure GitLab is running;
  Check the gitlab-shell configuration file:
  sudo -u git -H editor /opt/gitlab/gitlab-shell/config.yml
  Please fix the error above and rerun the checks.

Similarly, running gitlab-shell/bin/check directly fails in the same way

Internal API available: FAILED - 401 Unauthorized

and logs errors to gitlab-shell.log

time="2023-10-18T01:24:48Z" level=error msg="Internal API error" correlation_id=01HD0658K8PH16VMKH3AGWNGT6 duration_ms=152ns error="401 Unauthorized" method=GET status=401 url="http://unix/api/v4/internal/check"

I’ve tried various suggestions from the forums and elsewhere but none have helped. I can’t see any issues in the gitlab-shell config fie in /etc/gitlab-shell/config.yml. The gitlab_url is set to the following:

gitlab_url: "http+unix://%2Fopt%2Fgitlab%2Fgitlab%2Ftmp%2Fsockets%2Fgitlab-workhorse.socket"

If I intentionally add some garbage to this url the error changes to the following

Internal API available: FAILED - Internal API unreachable

and logs a reasonable message in gitlab-shell.log

time="2023-10-18T01:27:59Z" level=error msg="Internal API unreachable" correlation_id=01HD06B3G4563KM8PK6FV6DPDY duration_ms=0s error="Get \"http://unix/api/v4/internal/check\": dial unix /opt/gitlab/gitlab/tmp/sockets/gitlab-workhorse.socket.foobar: connect: no such file or directory" method=GET url="http://unix/api/v4/internal/check"

So at the moment I’m looking for debugging tips or other ideas on how to diagnose this. I suspect the fix will be simple once I find the problem. I haven’t found any particularly helpful messages in the various log files, they’re all relatively silent on these failures.

It’s definitely possible that I’ve missed specific steps on the upgrade paths (after totally forgetting about these and doing the upgrades above) but the gentoo ebuilds to prevent straying too far from the happy path so I’m hoping I haven’t missed anything terribly critical.

Some questions I have

  1. what is gitlab-shell/bin/check checking? is it the ssh connection (i.e. .ssh/authorized_keys issue) or the gitlab-shell binary executed by an ssh connection (i.e. command="/opt/gitlab/gitlab-shell/bin/gitlab-shell)?
  2. how does one construct a JWT token to test with curl to see if I get the same error? I’d like to be able to execute the curl here Internal API | GitLab to see if it fails in the same way.
  3. is this possibly a redis issue? I don’t see any redis related errors or errors in my redis log and I have run bundle exec rake cache:clear but that made no difference.
  4. what is likely returning the 401 Unauthorized error (gitlab-shell, gitlab-workhorse, the gitlab app, gitaly, etc.) and is there a log level I can increase somewhere to see why this is failing? I’m looking at this picture to try and understand how things all fit together GitLab architecture overview | GitLab

Many thanks for any help or suggestions!

Cheers,
Derek

2

The problem here turned out to be that I had inadvertently installed a newer version of gitlab-shell that no longer sends the Gitlab-Shared-Secret header.

1 Like