Docker can push to registry with client, but not in runner

How to Use GitLab Self-managed
, ,
1

I recently deployed the gitlab operator, and while everything is working fine, I can’t seem to push to a projects registry from a runner, but I can from my desktop client.

The output of the runner shows:

$ docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY && docker push $CI_REGISTRY_IMAGE
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded
Using default tag: latest
The push refers to repository [registry.domain.tld/user/project]
7331e3d44704: Preparing
3b19ae7151b0: Preparing
161f549396cc: Preparing
1dcb9a27d7a0: Preparing
4a9cf0691dda: Preparing
4a575cbea538: Preparing
958666fc7311: Preparing
3293ba39a105: Preparing
a1c2f058ec5f: Preparing
cc2447e1835a: Preparing
4a575cbea538: Waiting
958666fc7311: Waiting
3293ba39a105: Waiting
a1c2f058ec5f: Waiting
cc2447e1835a: Waiting
unauthorized: authentication required
Cleaning up project directory and file based variables
00:01
ERROR: Job failed: command terminated with exit code 1

My gitlab-ci.yml file:

stages:
- build

.docker_base:
  image: docker:24.0.5-dind
  services:
    - name: docker:24.0.5-dind
      command: ["--tls=false"]
  variables:
    DOCKER_DRIVER: overlay2
    DOCKER_HOST: tcp://docker:2375
    DOCKER_TLS_CERTDIR: ""
    FF_GITLAB_REGISTRY_HELPER_IMAGE: 1
  before_script:

build:
  extends:
    - .docker_base
  stage: build
  script:
  - nslookup registry.domain.tld
  - echo $CI_REGISTRY_IMAGE
  - >
    docker build
    --tag $CI_REGISTRY_IMAGE:latest
    --build-arg BUILD_DATE=$(date -u +'%Y-%m-%dT%H:%M:%SZ')
    --build-arg VCS_REF=$CI_COMMIT_SHORT_SHA
    ./Docker/
  #- docker tag $CI_REGISTRY_IMAGE:latest $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_NAME

  - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY && docker push $CI_REGISTRY_IMAGE

and when I try pushing from my client (already uploaded):

docker push registry.domain.tld/user/project:latest
The push refers to repository [registry.domain.tld/user/project]
31f3fb023e75: Layer already exists
2a2559c3b67f: Layer already exists
54404912ec5a: Layer already exists
a4568a5eac8d: Layer already exists
c039090a4db7: Layer already exists
4a575cbea538: Layer already exists
958666fc7311: Layer already exists
3293ba39a105: Layer already exists
a1c2f058ec5f: Layer already exists
cc2447e1835a: Layer already exists
latest: digest: sha256:8d1b4bf59e55e132bf09b58e1ce06767b2f19825b4e784e5bec996f8200fb369 size: 2409

i’ve tried every permutation of the magic token for the runner to access the registry, as well as switching to my user and a PAT that has write access (same one i use on my client), but it always says authentication required, despite the login IMMEDIATELY before saying it logged in just fine. I’ve checked to make sure DNS resolves correctly, I’ve tried using just “latest” for the tag, everything i can think of but no dice.

Gitlab version 16.4.1, latest version with latest version of the operator and runner.
If it matters, here’s my runner config:

runners:
  config: |
    [[runners]]
      [runners.kubernetes]
        image = "ubuntu:22.04"
        pull_policy = "always"
        privileged = true
        listen_address = "[::]:8043"
        namespace = "{{.Release.Namespace}}"
2

Hey,

Maybe I’m wrong, but aren’t you missing the tag in your push command (in the .gitlab-ci.yml)? E.g.

docker push $CI_REGISTRY_IMAGE:latest

P.S. I believe image should be non-dind one, and for services the dind one :slight_smile:

3
  1. Use $CI_COMMIT_REF_SLUG instead of $CI_COMMIT_REF_NAME
  2. I do not know what your current issue is, I’m not going to pretend I do.
  3. I would recommend that you do not run privileged runners, skip DinD altogether and use Kaniko for building container images and if you only need to “repost” images (copy from one registry to another) there is Skopeo. And if you are interested (shameless plug) I created a wrapper for Kaniko to make my life easier; build-oci.