Credential helper for AWS ECR docker login not working

I have followed the instructions here:
https://docs.gitlab.com/ee/ci/docker/using_docker_images.html#define-an-image-from-a-private-container-registry

But it gets this error (three times, then gives up):

Pulling docker image 558517226390.dkr.ecr.eu-west-1.amazonaws.com/regression-test/ddbuilder:20210303 
ERROR: Preparation failed: Error response from daemon: Get https://<myawsid>.dkr.ecr.eu-west-1.amazonaws.com/v2/regression-test/ddbuilder/manifests/20210303: no basic auth credentials (docker.go:198:5s)

I can confirm that I have an appropriate IAM user, and if I just place that user’s credentials in .aws/credentials then a “docker pull” as a normal user on my runner machine can pull the image from Amazon’s ECR registry.

I have built and installed the docker-credential-ecr-login binary as described, version 0.4.0, this is in the gitlab-runner user $PATH and it can run from the command line.

The AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY are set in the CI/CD variables, and they are unprotected.

DOCKER_AUTH_CONFIG is set to
{ “credsStore”: “ecr-login” }
I also put it in .docker/config.json, no help.

Not sure what next step to try. Is it possible to get more detailed logging from gitlab-runner (on Docker) ?
I set log_level = “debug” in config.toml, but no additional messages are sent.

My Gitlab runners are version 12.10.1.

I don’t have experience with AWS credential helper, but the log_level does not affect what you get in the Job output. GitLab Runner is using Syslog for logging so in default setups you get those logs in /var/log/messages on RHEL/CentOS or in /var/log/syslog in Debian/Ubuntu.

Thanks! Only relevant messages from log are here, doesn’t say much more.

Mar  4 07:33:03 jade3 dockerd[1309]: time="2021-03-04T07:33:03.196697294Z" level=info msg="Attempting next endpoint for pull after error: Get https://<awsaccount>.dkr.ecr.eu-west-1.amazonaws.com/v2/regression-test/ddbuilder/manifests/20210303: no basic auth credentials"
Mar  4 07:33:03 jade3 dockerd[1309]: time="2021-03-04T07:33:03.196756964Z" level=error msg="Handler for POST /v1.35/images/create returned error: Get https://<awsaccount>.dkr.ecr.eu-west-1.amazonaws.com/v2/regression-test/ddbuilder/manifests/20210303: no basic auth credentials"

Wondering what the “manifests” in the path is. The image id is ddbuilder:20210303.

very simply put manifest file tells Docker what should be downloaded. Image Manifest V 2, Schema 2 | Docker Documentation

Here are some hints about your error Troubleshooting Errors with Docker Commands When Using Amazon ECR - Amazon ECR

I wonder where do you specify Region? Usually this is in .aws/config. You can try specify it as AWS_REGION variable.
Google told me there should be a log file ~/.ecr/log/ecr-login.log as well.

Thanks!

I have now, by monitoring ~gitlab-runner/.ecr/log/ecr-login.log discovered that it gets written to if I call from the command line (as gitlab-runner):

sudo docker pull <image>

or if I call

docker-credential-ecr-login list

directly.

But when the gitlab-runner is triggered from Gitlab CI, nothing gets written to the ECR log.
It looks like the actual gitlab runner process is for some reason not getting that config i.e. from DOCKER_AUTH_CONFIG, telling it to use the ECR credentials helper.

DOCKER_AUTH_CONFIG is set in my CI variables, and is not protected. It contains:

{ "credsStore": "ecr-login" }

Any way to get the runner to report what variables it has set when it runs?

I think gitlab-runner cannot find or execute the docker-credential-ecr-login

Seems to be available, executable by gitlab-runner user:

gitlab-runner@jade3:~$ which docker-credential-ecr-login 
/usr/bin/docker-credential-ecr-login
gitlab-runner@jade3:~$ ls -lrt /usr/bin/docker-credential-ecr-login
-rwxr-xr-x 1 root root 8482816 Oct 25 09:00 /usr/bin/docker-credential-ecr-login
gitlab-runner@jade3:~$ docker-credential-ecr-login 
Usage: docker-credential-ecr-login <store|get|erase|list|version>