Gitlab runner gets "HTTP Basic: Access denied" when project is set to internal/private on self hosted 16.4.2

Problem

I have problems with the GitLab Runners with Shell executor I have set up:

  • When the project visibility is set to public, they work fine.
  • When I set the project to visibility internal or private, I get HTTP Basic : Access Denied when the runner tries to clone the repository from GitLab before starting to run the pipeline.
    • I also have Docker runners with Podman as container engine. These continue to be able to clone the repo even when the project is set to internal or private. However, for jobs which tries to do any git operations as part of the job, they also fail with HTTP Basic : Access Denied. (For example, I have a job which does git fetch origin master. That fails when the project visibility is set to other states than public.

Environment

This is on a self managed GitLab Enterprise Edition on version 16.4.2. It is an omibus install, RPM package on AlmaLinux 8.8 which is in an air-gapped environment. It is connected to LDAP, and Kerberos sign in is configured. Users sign in using Kerberos.
The runners are on version 16.4.1, installed using the RPM package on AlmaLinux 8.8.

Troubleshooting steps taken

  • I tried the suggestion in this post regarding using clone-url. Did not help.
  • Tried to install a different version of the GitLab runner package on a runner: 15.11.1. Did not help
  • Rebooting the GitLab server and the runners. Did not help.
  • Logged in to the gitlab runner host using the account gitlab-runner, and tried to clone using https and a private access token. This works fine. However, when I paste the exact same command string inside a job, it fails with HTTP Basic : Access Denied.

Logs

Since this is inside an offline, air-gapped environment I can not easily paste logs. But here is an outline of the GitLab runner logs I see in GitLab’s web UI:

  1. Running with gitlab-runner 16.4.1 (d89a789a)
  2. on runner-01…
  3. Resolving secrets
  4. Preparing the “shell” executor
  5. Using Shell (bash) executor…
  6. Preparing environment
  7. Running on runner01…
  8. Getting source from Git repository
    12 Fetching changes with git depth set to 20…
  9. Reinitialized existing Git repository in /home/gitlab-runner/builds/2aVCeBoN/0/groupname/projectname/.git/
  10. remote: HTTP Basic: Access denied. The provided password or token is incorrect or your account has 2FA enabled and you must use a personal access token instead of a password. See Troubleshooting Git | GitLab
  11. fatal: Authentication failed for ‘https://gitlab.domain.se/group-name/project.git
  12. Cleaning up project direcotry and file based variables
  13. ERROR: Job failed: exist status 1

The logs above was manually typed in by me looking at the actual log in the offline network. May contain typos, but I have intended not to. I have changed the URL of our gitlab server, and the name of the group and project in the text above.

Did you ever resolve this?
I have just run in to the exact same problem.