DOCKER_AUTH_CONFIG not used by runner

GitLab CI/CD
1

I have a gitlab server and a gitlab-runner with docker executor. I have created a global variable named DOCKER_AUTH_CONFIG and created this example .gitlab-ci.yml file in a repository:

build:
    variables:
        CI_DEBUG_TRACE: "true"
    script:
        - export

This job runs fine, and in the job output, I can see that DOCKER_AUTH_CONFIG variable is there. It looks like this:

Running with gitlab-runner 16.2.0 (782e15da)
  on ***** *****, system ID: r_*****
Preparing the "docker" executor
00:03
Using Docker executor with image ruby:2.7 ...
Pulling docker image ruby:2.7 ...
Using docker image sha256:3cb86b8c626861ba5167462228a0035db6b2deae893f3e8fafcf256d9d3abdaa for ruby:2.7 with digest ruby@sha256:2347de892e419c7160fc21dec721d5952736909f8c3fbb7f84cb4a07aaf9ce7d ...
Preparing environment

... more lines here...

declare -x DOCKER_AUTH_CONFIG="{
	\"auths\": {
		\"my.private.registry.com\": {
			\"auth\": \"*******************=\"
		}
}"

... more lines here...

+ exit 0
Job succeeded

However, if I try to use an image from the private repository like this:

build:
    variables:
        CI_DEBUG_TRACE: "true"
    image: my.private.registry.com/my-private-image:latest
    script:
        - ./build_backend.sh

Then I get this job output:

Running with gitlab-runner 16.2.0 (782e15da)
  on ******* *******, system ID: r_*******
Preparing the "docker" executor
00:01
Using Docker executor with image my.private.registry.com/my-private-image:latest ...
Pulling docker image my.private.registry.com/my-private-image:latest ...
WARNING: Failed to pull image with policy "always": Error response from daemon: Head "https://my.private.registry.com/v2/my-private-image/manifests/latest": no basic auth credentials (manager.go:237:0s)
ERROR: Job failed: failed to pull image "my.private.registry.com/my-private-image:latest" with specified policies [always]: Error response from daemon: Head "https://my.private.registry.com/v2/my-private-image/manifests/latest": no basic auth credentials (manager.go:237:0s)

Gitlab version is 16.1.2, Gitlab runner version is 16.2.0.

I have another gitlab installation where it works fine. On that installation, the job begins like this:

Preparing the "docker" executor
00:01
Using Docker executor with image another.private.registry.com/another-image:latest ...
Authenticating with credentials from $DOCKER_AUTH_CONFIG
Pulling docker image another.private.registry.com/another-image:latest ...

Interestingly, the CI_DEBUG_TRACE setting does nothing if the image cannot be pulled. It only works after the image has been pulled and the job runner has been started. This is why I used the first example config, and double checked that the DOCKER_AUTH_CONFIG variable is present, and it is correct. The only difference I see between these two installations is that the one that works has the gitlab-runner and the gitlab on the same computer, whereas the bad one uses two different computers.

2

make sure there is no typo in the variable and also that the DOCKER_AUTH_CONFIG is not overwritten on some other level like on runner or project.

3

Job log was copied out from the real job log, and then host names replaced by hand. You can see that there’s no typo. It’s value is correct, it is a JSON object. Even if the password was incorrect, the “Authenticating with credentials…” line should appear, right?

4

hello, did you manage to resolve your issue ?

i have the exact same behavior on my side ( $DOCKER_AUTH_CONFIG not appearing on the faulty runner when requesting the private registry).

On my setup, the 2 runners are on 2 separate VM located on another network from the private registry. The only thing that change is the version of gitlab-runner used for each one.