Troubles pulling from a private DockerHub repo with DOCKER_AUTH_CONFIG on Windows runner

Troubles pulling from a private DockerHub repo with Windows runner

I’ve been able to successfully run my gitlab-ci pipeline when my DockerHub repo was public - but not so much since i’ve switched the repo to private.

The pipeline error i’m receiving:

Running with gitlab-runner 12.7.0 (58272c27)
  on multi-purpose mnyax9cs
Using Docker executor with image my_private_repo_name/project:tag ...
Pulling docker image my_private_repo_name/project:tag ...
ERROR: Job failed: Error response from daemon: pull access denied for my_private_repo_name/project, repository does not exist or may require 'docker login': denied: requested access to the resource is denied (executor_docker.go:188:0s)

From the error message it appears to be an authentication to dockerhub issue or I’m not fully qualifying the location of my dockerhub repo now that it’s private or maybe i’m missing a port number.

My environment:

• GitLab ee 12.7.0 on Ubuntu 18.04
• Runner host machine is running Win 1809
  • gitlab-runner.exe installed as a serivce - docker-windows executor - runner version 12.7.0
  • Docker Desktop for Windows installed [v 2.2.0.0 (42247)]

My gitlab-ci.yml:

image: my_private_repo_name/project:tag
test:
  artifacts:
    paths:
    - c:\builds\username\nghello_world_node\dist\
  script:
# Check for NPM updates
    - npm -g install npm
# Install Node dependencies
    - npm install
# Compile Angular Code
    - npm run build

Things I’ve tried thus far:

• Following these instructions to add a DOCKER_AUTH_CONFIG variable to the runner’s config.toml file:

  [[runners]]
  environment = ["DOCKER_AUTH_CONFIG={\"auths\":{\"my_private_repo_name/project\":{\"auth\":\"base64_username:password_string\"}}}"]
  

Where base64_username:password_string is a base64 encoding of my dockerhub “username:password” - using this section of gitlab documentation.

• I’ve tried adding a DOCKER_AUTH_CONFIG variable to my project’s CI/CD settings (gitlab project>Settings>CI/CD>Variables - following these instructions to create a docker config.json file and add the contents of this file as a variable.

image945×403 10.6 KB

Value: 
{
    "auths": {
        "my_private_repo_name/project": {
            "auth": "(base64_username:password_string)"
        }
    }
}

• I’ve also tried referencing the private DockerHub repo more specifically/more uniquely when specifying the private repo location - for instance, the image value in gitlab-ci.yml file:

image: https://hub.docker.com/repository/docker/my_private_repo_name/project:tag

or

image: registry.hub.docker.com/my_private_repo_name/project:tag

I’ve also tried adjusting the same repo location in the runner’s config.toml and in the DOCKER_AUTH_CONFIG variable I configured - all to no avail.

I’m wondering if anyone else has run into this issue or has noticed anything that i’m missing.
Any insight is much appreciated!

I recommend that you remove the entry from the runner config. Everybody with access to it can use your credentials (when it works).

Add a pre-build to your gitlab-ci.yml and echo the content of the env. var. to a new config.json file. Replace the Linux based syntax for your Windows shell.

image: my_private_repo_name/project:tag
test:
  artifacts:
    paths:
    - c:\builds\username\nghello_world_node\dist\
  pre-build:
    - mkdir -p $HOME/.docker
    - echo $DOCKER_AUTH_CONFIG > $HOME/.docker/config.json
  script:
# Check for NPM updates
    - npm -g install npm
# Install Node dependencies
    - npm install
# Compile Angular Code
    - npm run build

Thanks for the response @nightman68. I’ve removed the DOCKER_AUTH_CONFIG variable from the runner’s config file - thanks for that recommendation.

Also I modified my gitlab-ci.yml file to match your example and made sure DOCKER_AUTH_CONFIG was a defined variable within my gitlab project. (settings>CI/CD>Variables)

Lint threw a yml syntax error - wasn’t happy with the pre-build task nested within the test job. So I moved pre-build just above the test job:

image: my_private_repo_name/project:tag
pre-build:
  script:
  - mkdir -p c:\Users\username\.docker
  - echo $DOCKER_AUTH_CONFIG > c:\Users\username\.docker\config.json
test:
  artifacts:
    paths:
    - c:\builds\username\nghello_world_node\dist\
  script:
# Check for NPM updates
    - npm -g install npm
# Install Node dependencies
    - npm install
# Compile Angular Code
    - npm run build

I received the same error as a before:

Running with gitlab-runner 12.7.0 (58272c27)
  on multi-purpose mnyax9cs
Using Docker executor with image my_private_repo_name/project:tag ...
Pulling docker image my_private_repo_name/project:tag ...
ERROR: Job failed: Error response from daemon: pull access denied for my_private_repo_name/project, repository does not exist or may require 'docker login': denied: requested access to the resource is denied (executor_docker.go:188:0s)

I was able to browse to the location of the config.json file on the Windows runner host:

c:\Users\username\.docker\config.json

Here is the contents of that config.json file after I ran the pipeline:

{
	"auths": {
		"https://index.docker.io/v1/": {},
		"my_private_repo_name/project": {}
	},
	"HttpHeaders": {
		"User-Agent": "Docker-Client/19.03.5 (windows)"
	},
	"credsStore": "desktop",
	"stackOrchestrator": "swarm"
}

Are there any visible issues with this file?

I also tired directly modifying this config.json file to add my base64 credentials and then re-ran the pipeline. This caused my docker desktop session on my Win runner host to force logout and upon the next docker login I got the following error:

PS C:\GitLab-Runner> docker login
WARNING: Error loading config file: C:\Users\username\.docker\config.json: illegal base64 data at input byte 0
Authenticating with existing credentials...
Login Succeeded

Any other thoughts on how to access a private dockerhub repo ?

Kind Regards,

Ok my mistake sorry, wrong keyword. Try this:

image: my_private_repo_name/project:tag
test:
  artifacts:
    paths:
    - c:\builds\username\nghello_world_node\dist\
  before_script:
  - mkdir -p c:\Users\username\.docker
  - echo $DOCKER_AUTH_CONFIG > c:\Users\username\.docker\config.json
  script:
# Check for NPM updates
    - npm -g install npm
# Install Node dependencies
    - npm install
# Compile Angular Code
    - npm run build

My config.json on my MAC looks like this

{
	"auths": {
		"REGISTRY_01": {
			"auth": "THE ENCRYPTED TOKEN"
		},
		"REGISTRY_02": {
			"auth": "THE ENCRYPTED TOKEN"
		}
	}
}

I recommend that you’re not use the home directory of a “normal” user. Settings made by this user manually are overwritten each time the job is running. The runner has a parameter which can be used to define a dedicated directory for HOME.

No worries - thanks for the response. Ok so I’m a little confused with how this yml file is interpreted.

From my understanding it seems like the before_script step within the test job would be executed by the runner inside the docker container once it’s been pulled from the private dockerhub repo - however, the pipeline job is not able to make it that far since the runner is not able to pull the image: from the private dockerhub repo.

It’s almost like I need to define a docker authentication variable before the pipeline even tries to pull an image from the private dockerhub repo. Maybe i’m misunderstanding how this before_script: step works.

I modified my gitlab-ci.yml file anyway to try your recommendation, but I received the same results:

Running with gitlab-runner 12.7.0 (58272c27)
on multi-purpose gvbLXrk3
Using Docker executor with image my_private_repo_name/project:tag ... 00:00
Pulling docker image my_private_repo_name/project:tag ...
ERROR: Job failed: Error response from daemon: pull access denied for my_private_repo_name/project, repository does not exist or may require 'docker login': denied: requested access to the resource is denied (executor_docker.go:188:0s)

Any other thoughts?

Kind Regards,

The details about about before_script you’ll find here. I use this for Docker on Linux and it works…

Have you removed this "credsStore": "desktop" from the env. var.?

I’m not familiar with the docker-windows executor. Have you checked the system log of the Windows box for runner messages? Try to enable debugging for CI, details are here.

I would add a line which prints the content of the “echoed” config file that you can see in the in CI output.

One more thing: may be this helps debugging the pull.

Thanks for the reply. I removed “credsStore”: “desktop” from config.json file and from what I can tell - it seems to get added back each time you login into the Docker client. I also noticed that every time I save the base64 creds into the config.json file and rerun the pipeline, the docker desktop client force logs off - weird.

Enabled CI_DEBUG_TRACE within my gitlab-ci.yml file, although the debugging console entries begin after the pipeline fails to pull from the private DockerHub repo - this will be really useful in the future though.

The Docker client logs look the most promising however. I’ll keep trying things and sharing what I’ve tried to this post.

Thanks,

You really should configure the runner using a dedicated directory what is independent from your home account. No idea what Docker desktop is doing in the background…

Good luck!

I recommend you to read following part of GitLab docks:

I suppose issue is in your configured variable DOCKER_AUTH_CONFIG.
Try to configure it like mentioned in GitLab docks. Following way worked for me:

{
	"auths": {
		"https://index.docker.io/v1/": {
			"auth": "hashed value of your credentials"
		}
	}	
}

Don’t forget to change following row with your value “auth”: “hashed value of your credentials”.
If you configured this variable on GitLab GUI interface, then you don’t need to mention this variable additionally in your yml file.