Issue connection docker registry

Dear fellow gitlab users,

I want to connect to my docker registry and pull the images from there.

the setup

My selfhosted Gitlab is running inside of a Docker Container (gitlab/gitlab-ce:13.9.3-ce.0) on host A (10.0.0.130), my runners are also there (docker runner).

I want to get the images on host B

I have a third server host C (10.0.0.120) which terminates SSL. Here runs an apache revers proxy with Let's Encrypt certs. This is setup to allow external access to gitlab.

the problem

In my logs/registry/current is the following error:

On host B I get the following error (I tried it directly with and without reverseproxy, I also added it as insecure registry):
/v2/: denied: access forbidden

debuging with curl

I found an description how to test it directly with curl - there I get (I think) a proper token (I checked it on jwt.io)

However also the correct token does not work.

Config

This is my config/gitlab.rb

# docker registry
registry['enable'] = true
gitlab_rails['registry_enabled'] = true

# http since ssl is done by reverse proxy; port is 1443 since this is passed through by docker
registry_external_url 'http://******************ace:1443'


# https://docs.gitlab.com/ee/administration/packages/container_registry.html
gitlab_rails['registry_api_url'] = "https://******************ace:1443"

gitlab_rails['registry_host'] = "******************ace"
gitlab_rails['registry_port'] = "1443"

When I set registry_external_url to https gitlab can’t start since it can’t find a certificate

another issue

another issue arised since I started playing around an restarting gitlab multiple times (or just reconfiguring it). I’m not sure if this has anything to do with the other issue here.
There is an closed(??) issue regarding this error: Something went wrong while fetching the repository list While accessing container registry in UI (#295663) · Issues · GitLab.org / GitLab · GitLab

Things that do work

However the gitlab docker runners can push to 10.0.0.130 (with insecure registry set). This works with Dind and Kaniko

My Thoughts

Since I can get a token and the runners can access the registry there should be no problem with the registry itself nor with token creation.
It seems to be a permission problem → therefor I tested another created token with all rights and also tested my admin user which is the project Maintainer and Group Owner

It also worked in an older version of gitlab (I think 11.1.4-ce.0) with and without the reverse proxy.

I would appreciate if someone could point me in the correct direction.

Some additional things I found out today:

jwt token was wrong

as the answer of the second curl request told us the token is not accepted
I queried the token for the “wrong project(path)” - I had the image name included like in an example I found
This is the correct call:

ci_token=$(curl -sS --user "test:nXc5yy-_ZDLDH_Md-W99" "https://<<gitlab-tld>>/jwt/auth?service=container_registry&scope=repository:<<group>>/<<project>>:push,pull" | cut -d'"' -f 4)
echo $ci_token
curl -H "Authorization: Bearer $ci_token" https://<<registry-tld>>:1443/v2/<<group>>/<<project>>/manifests/latest

realm in settings of registry is wrong

as found here: Registry generated conf is incorrect. realm line (#41375) · Issues · GitLab.org / GitLab FOSS · GitLab
my registry/config.yml was wrong:

auth:
  token:
    realm: httpS://<<gitlab-tld>>/jwt/auth
    service: container_registry
    issuer: omnibus-gitlab-issuer
    rootcertbundle: /var/opt/gitlab/registry/gitlab-registry.crt
    autoredirect: false
validation:
  disabled: true

the Sin httpS was missing, it was only http (I added the lowercase S)

and finally fixed it by sv restart registry however this is a temporary solution since the settings will be overwritten automatically by gitlab

now login & pull registry works from Host B