Own hosted GitLab CI | Pushing registry image fails

Hi,

I’ve setup my own hosted GitLab with Docker to automatically build images and upload to GitLab. It seems to working well … until the end:

[...]
$ docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded
$ echo $CI_REGISTRY
gitlab.example.com
$ echo $CI_REGISTRY_USER
gitlab-ci-token
$ echo $CI_REGISTRY_IMAGE
registry.example.com/user/apache
$ docker build --pull -t "$CI_REGISTRY_IMAGE" .
[...]
Successfully built fd3250205647
Successfully tagged registry.example.com/user/apache:latest
$ docker push "$CI_REGISTRY_IMAGE"
The push refers to repository [registry.example.com/user/apache]
796886f67eb9: Preparing
[..]
no basic auth credentials
ERROR: Job failed: exit code 1

What I did
docker-compose.yml

  gitlab:
    image: 'gitlab/gitlab-ce:latest'
    container_name: gitlab
    restart: unless-stopped
    depends_on:
      - traefik
      - gitlab-runner
    hostname: 'gitlab.example.com'
    environment:
      GITLAB_OMNIBUS_CONFIG: |
        external_url 'https://gitlab.example.com'
        registry_external_url 'https://registry.example.com'
        [...]
    volumes:
        - './gitlab/config:/etc/gitlab'
        - './gitlab/logs:/var/log/gitlab'
        - './gitlab/data:/var/opt/gitlab'
    labels:
      [...]
      - "traefik.frontend.rule=Host:gitlab.example.com"
    networks:
      - traefik_proxy

  gitlab-runner:
    image: 'gitlab/gitlab-runner:latest'
    container_name: gitlab-runner
    restart: unless-stopped
    volumes:
        - './gitlab-runner/config:/etc/gitlab-runner'
        - /var/run/docker.sock:/var/run/docker.sock
    networks:
      - traefik_proxy

  registry:
    image: registry:2
    container_name: registry
    restart: unless-stopped
    ports:
      - 5000:5000
    environment:
      REGISTRY_AUTH: htpasswd
      REGISTRY_AUTH_HTPASSWD_PATH: /auth/htpasswd
      REGISTRY_AUTH_HTPASSWD_REALM: Registry Realm
    volumes:
      - ./registry/data:/var/lib/registry
      - ./registry/certs:/certs
      - ./registry/auth:/auth
    labels:
      [...]
      - "traefik.frontend.rule=Host:registry.example.com"
    networks:
      - traefik_proxy

my .gitlab-ci.yml is nearly the template one:

build-master:
  image: docker:latest
  stage: build
  services:
    - docker:dind
  variables:
    DOCKER_DRIVER: overlay2
    # Create the certificates inside this directory for both the server
    # and client. The certificates used by the client will be created in
    # /certs/client so we only need to share this directory with the
    # volume mount in `config.toml`.
    DOCKER_TLS_CERTDIR: "/certs"
  before_script:
    - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY
  script:
    - echo $CI_REGISTRY
    - echo $CI_REGISTRY_USER
    - echo $CI_REGISTRY_IMAGE
    - docker build --pull -t "$CI_REGISTRY_IMAGE" .
    - docker push "$CI_REGISTRY_IMAGE"
  only:
    - master

Setting up a runner
cat ./gitlab-runner/config/config.toml

concurrent = 1
check_interval = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "Docker runner"
  url = "https://gitlab.example.com/"
  token = "<token>"
  executor = "docker"
  [runners.custom_build_dir]
  [runners.docker]
    tls_verify = false
    image = "docker:19"
    privileged = true
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/certs/client", "/cache"]
    shm_size = 0
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]

After restarting all services I would expect to login with my GitLab credentials:

docker login -u user registry.example.com

but this also fails when I’m using a access token with scope rite_repository
docker logs registry

time="2019-09-11T21:32:53.21327076Z" level=warning msg="error authorizing context: basic authentication challenge for realm "Registry Realm": invalid authorization credential"
time="2019-09-11T21:32:53.263548062Z" level=error msg="error authenticating user "user": authentication failure"
time="2019-09-11T21:32:53.264270314Z" level=warning msg="error authorizing context: basic authentication challenge for realm "Registry Realm": authentication failure" 

I’m not sure if it is really required to login successfully. But the above mentioned push shall work. How can I solve this issue?

Thanks a lot to you!

I’ve tried again to run the job - getting the same issue.

The push refers to repository [registry.example/user/apache]
[...]
no basic auth credentials
ERROR: Job failed: exit code 1

But now I’ve had a look at registry’s logs:

time="2019-09-12T21:21:37.910020511Z" level=info msg="PurgeUploads starting: olderThan=2019-09-05 21:21:37.897525856 +0000 UTC m=-602879.579789293, actuallyDelete=true"
time="2019-09-12T21:21:37.923001528Z" level=info msg="Purge uploads finished.  Num deleted=0, num errors=1"
time="2019-09-12T21:21:37.923937899Z" level=info msg="Starting upload purge in 24h0m0s" go.version=go1.11.2 instance.id=a38e55e2-d778-4001-98fb-849337438e54 service=registry version=v2.7.1
time="2019-09-12T21:23:41.896988177Z" level=warning msg="error authorizing context: basic authentication challenge for realm "Registry Realm": invalid authorization credential" go.version=go1.11.2 http.request.host=registry.example.com http.request.id=25f0f0e6-620c-42f1-9eee-03118132757b http.request.method=GET http.request.remoteaddr=172.23.0.1 http.request.uri="/v2/" http.request.useragent="docker/19.03.2 go/go1.12.8 git-commit/6a30dfca03 kernel/4.9.0-9-amd64 os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.2 \(linux\))"
172.24.0.8 - - [12/Sep/2019:21:23:41 +0000] "GET /v2/ HTTP/1.1" 401 87 "" "docker/19.03.2 go/go1.12.8 git-commit/6a30dfca03 kernel/4.9.0-9-amd64 os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.2 \\(linux\\))"

I’m pretty sure that there went something wrong in the registry configuration. Since it is required to give the certificate to registry, I have extracted the cert and key from traefik to gitlab’s volume which is pointing to /etc/gitlab/ssl/.
The question is: Is this the correct way to provide the traefik generated certificate to the gitlab?