DOCKER_AUTH_CONFIG not used by runner

I have a gitlab server and a gitlab-runner with docker executor. I have created a global variable named DOCKER_AUTH_CONFIG and created this example .gitlab-ci.yml file in a repository:

build:
    variables:
        CI_DEBUG_TRACE: "true"
    script:
        - export

This job runs fine, and in the job output, I can see that DOCKER_AUTH_CONFIG variable is there. It looks like this:

Running with gitlab-runner 16.2.0 (782e15da)
  on ***** *****, system ID: r_*****
Preparing the "docker" executor
00:03
Using Docker executor with image ruby:2.7 ...
Pulling docker image ruby:2.7 ...
Using docker image sha256:3cb86b8c626861ba5167462228a0035db6b2deae893f3e8fafcf256d9d3abdaa for ruby:2.7 with digest ruby@sha256:2347de892e419c7160fc21dec721d5952736909f8c3fbb7f84cb4a07aaf9ce7d ...
Preparing environment

... more lines here...

declare -x DOCKER_AUTH_CONFIG="{
	\"auths\": {
		\"my.private.registry.com\": {
			\"auth\": \"*******************=\"
		}
}"

... more lines here...

+ exit 0
Job succeeded

However, if I try to use an image from the private repository like this:

build:
    variables:
        CI_DEBUG_TRACE: "true"
    image: my.private.registry.com/my-private-image:latest
    script:
        - ./build_backend.sh

Then I get this job output:

Running with gitlab-runner 16.2.0 (782e15da)
  on ******* *******, system ID: r_*******
Preparing the "docker" executor
00:01
Using Docker executor with image my.private.registry.com/my-private-image:latest ...
Pulling docker image my.private.registry.com/my-private-image:latest ...
WARNING: Failed to pull image with policy "always": Error response from daemon: Head "https://my.private.registry.com/v2/my-private-image/manifests/latest": no basic auth credentials (manager.go:237:0s)
ERROR: Job failed: failed to pull image "my.private.registry.com/my-private-image:latest" with specified policies [always]: Error response from daemon: Head "https://my.private.registry.com/v2/my-private-image/manifests/latest": no basic auth credentials (manager.go:237:0s)

Gitlab version is 16.1.2, Gitlab runner version is 16.2.0.

I have another gitlab installation where it works fine. On that installation, the job begins like this:

Preparing the "docker" executor
00:01
Using Docker executor with image another.private.registry.com/another-image:latest ...
Authenticating with credentials from $DOCKER_AUTH_CONFIG
Pulling docker image another.private.registry.com/another-image:latest ...

Interestingly, the CI_DEBUG_TRACE setting does nothing if the image cannot be pulled. It only works after the image has been pulled and the job runner has been started. This is why I used the first example config, and double checked that the DOCKER_AUTH_CONFIG variable is present, and it is correct. The only difference I see between these two installations is that the one that works has the gitlab-runner and the gitlab on the same computer, whereas the bad one uses two different computers.

make sure there is no typo in the variable and also that the DOCKER_AUTH_CONFIG is not overwritten on some other level like on runner or project.

Job log was copied out from the real job log, and then host names replaced by hand. You can see that there’s no typo. It’s value is correct, it is a JSON object. Even if the password was incorrect, the “Authenticating with credentials…” line should appear, right?