GitLab Runner 15.9.1 "Access Denied" performing any git action with valid credentials

How to Use GitLab Self-managed
,
1

Our team recently migrated from Gitlab 15.5 to 15.9.1,

Unfortunately, we did not test our pipelines before moving data into the environment despite successfully performing authenticated git actions in various groups and projects. Every pipeline we run is immediately hit with the following error when the runner attempts to check-out the git project:

remote: HTTP Basic: Access denied. The provided password or token is incorrect or your account has 2FA enabled and you must use a personal access token instead of a password
fatal: authentication failed for '<my_project_url.git>'

Yes, our instance has LDAP SSO enabled however this is the CI_JOB_TOKEN failing to check out the git repo at the beginning of the pipeline.

As a test, in my .gitlab-ci.yml we tried the following to rule out an issue with the CI_JOB_TOKEN:

variables:
     GIT_STRATEGY: none
     GIT_CHECKOUT: "false"

This allowed a stage to complete a step of just running “pwd” on my shell based runner to print the current directory.

However in my next step I tried just performing a git clone with my username+PAT which successfully worked via linux CLI earlier:

test-pipeline
     stage: test
     script:
          - git clone "https://<my_username>:<my_token>@gitlab.my.domain/myprojecturl.git"

This produced the same “Access Denied” error despite the fact that I can run the same exact line on my local linux machine the shell based runner is on without the authentication error.

Other things I have tried:

  • We tried group & project runners from various systems (RHEL, Mac, Ubuntu)
  • We tried project access tokens, personal access tokens, and the CI_JOB_TOKEN and they all fail when done via runner
  • Whitelisted the runner IP in the GitLab UI
  • I tried running a CI job as a non-LDAP/SSO user
  • I have ensured HTTP basic authentication is enabled via Gitlab Admin UI, and again I reiterate, we can git clone, checkout, commit via CLI just fine using the same exact command+credentials that the pipeline/runner is trying to use in our 2nd test.
  • We have tried different runner versions (besides 15.9.1) both older and newer
  • turning the repository from “internal” to “public” works but is kind of obvious and not particularly helpful in diagnosing our problem

The common problem seems to be that the runner is screwing up git authentication somehow.

This has brought our entire program’s migration to a standstill and we have had no luck with Gitlab support. I honestly can’t remember the last time I got stumped on something this odd and I’m running low on luck. Greatly appreciate any insight.

1 Like
2

Hi,
not sure if this is relevant for your particular issue, but i just fixed mine, very similar, by forcing http clone url in settings to https. I’m using haproxy in front of gitlab for ssl-offload and i’m also forcing http connections to https. Hence, gitlab-ci-token user/pass were lost in this redirect, since they are passed in the url.
I hope it’ll help

3

@brendts How did you solve this issue? I seem have the same problem.

4

I can’t remember but this is how I’m using the gitlab job token successfully currently:

git clone https://gitlab-ci-token:${CI_JOB_TOKEN}@gitlab.com/<my-group>/$GL_PROJECT_NAME.git

I think just hard-coding the username as gitlab-ci-token worked.