I’m in the process of upgrading my home gitlab installation after neglecting it for a year and a half and at the moment I’m having some fun with gitlab-shell.
This is running on gentoo using the gitlab overlay from
Given that I haven’t touched this installation for a long time it’s turning out to be somewhat complicated. I have successfully upgraded from 14.9.2 → 14.9.5 → 14.10.4 → 15.0.0 and everything seemed to be fine through these stages. The web ui was happy and
bundle exec rake gitlab:check was all green. The latest upgrade I’ve done was from 15.0.0 → 15.0.5 and now
bundle exec rake gitlab:check fails with the following
Checking GitLab Shell ... GitLab Shell: ... GitLab Shell version >= 14.3.0 ? ... OK (14.13.0) Running /opt/gitlab/gitlab-shell/bin/check Internal API available: FAILED - 401 Unauthorized gitlab-shell self-check failed Try fixing it: Make sure GitLab is running; Check the gitlab-shell configuration file: sudo -u git -H editor /opt/gitlab/gitlab-shell/config.yml Please fix the error above and rerun the checks.
gitlab-shell/bin/check directly fails in the same way
Internal API available: FAILED - 401 Unauthorized
and logs errors to
time="2023-10-18T01:24:48Z" level=error msg="Internal API error" correlation_id=01HD0658K8PH16VMKH3AGWNGT6 duration_ms=152ns error="401 Unauthorized" method=GET status=401 url="http://unix/api/v4/internal/check"
I’ve tried various suggestions from the forums and elsewhere but none have helped. I can’t see any issues in the gitlab-shell config fie in
gitlab_url is set to the following:
If I intentionally add some garbage to this url the error changes to the following
Internal API available: FAILED - Internal API unreachable
and logs a reasonable message in
time="2023-10-18T01:27:59Z" level=error msg="Internal API unreachable" correlation_id=01HD06B3G4563KM8PK6FV6DPDY duration_ms=0s error="Get \"http://unix/api/v4/internal/check\": dial unix /opt/gitlab/gitlab/tmp/sockets/gitlab-workhorse.socket.foobar: connect: no such file or directory" method=GET url="http://unix/api/v4/internal/check"
So at the moment I’m looking for debugging tips or other ideas on how to diagnose this. I suspect the fix will be simple once I find the problem. I haven’t found any particularly helpful messages in the various log files, they’re all relatively silent on these failures.
It’s definitely possible that I’ve missed specific steps on the upgrade paths (after totally forgetting about these and doing the upgrades above) but the gentoo ebuilds to prevent straying too far from the happy path so I’m hoping I haven’t missed anything terribly critical.
Some questions I have
- what is gitlab-shell/bin/check checking? is it the ssh connection (i.e.
.ssh/authorized_keys issue) or the gitlab-shell binary executed by an ssh connection (i.e.
- how does one construct a JWT token to test with curl to see if I get the same error? I’d like to be able to execute the curl here Internal API | GitLab to see if it fails in the same way.
- is this possibly a redis issue? I don’t see any redis related errors or errors in my redis log and I have run
bundle exec rake cache:clearbut that made no difference.
- what is likely returning the
401 Unauthorizederror (gitlab-shell, gitlab-workhorse, the gitlab app, gitaly, etc.) and is there a log level I can increase somewhere to see why this is failing? I’m looking at this picture to try and understand how things all fit together GitLab architecture overview | GitLab
Many thanks for any help or suggestions!