Container registry with reverse proxy throws < token errors on auth

How to Use GitLab
,
1

Hi @dnsmichi & @nightman68 ,

I’m having some issue with the Gitlab Container Registry.

Checklist

  1. Gitlab UI: https://gitlab.mydomain.com - Works as expected. :white_check_mark:

  2. Git SSH Clone/Push/Pull/etc… Domain: git.mydomain.com - Works as expected. :white_check_mark:

  3. Container registry: https://acr.mydomain.com - Not working. :x:

    $ docker login acr.mydomain.com -u my_username -p my_password
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
Error response from daemon: Get https://acr.mydomain.com/v2/: unable to decode token response: invalid character '<' looking for beginning of value

    I get the same error even if i try the following:

    1. I enable all rights in acces token including sudo

    2. I enable only read_registry, write_registry

    3. I use the real password of the Gitlab User and not an access token

    May not be releveant at all, but the curious part is why the extra, ‘:’ added in the end of ‘https://acr.mydomain.com/v2/:’. If I go to https://acr.mydomain.com/v2/ using a browser, I can actually see reasonable response:

    {
  "errors": [
    {
      "code": "UNAUTHORIZED",
      "message": "authentication required",
      "detail": null
    }
  ]
}

    The following route seems to work reasonably too:

    curl https://acr.mydomain.com/v2/_catalog

    {
  "errors": [
    {
      "code": "UNAUTHORIZED",
      "message": "authentication required",
      "detail": [
        { "Type": "registry", "Class": "", "Name": "catalog", "Action": "*" }
      ]
    }
  ]
}

    I have a feeling It’s just that ‘:’ in the end that’s causing the issue. Anyone know how it could have been introduced?

    If I browse for https://acr.mydomain.com/v2/: using the browser, I get an 404 page not found. Which kinda makes sense, because why would a : be there? But it seems it’s docker login that’s adding the extra colon.

    Or I totally really may have missed something else here and that the colon is probably really only there for error message formatting. :sweat_smile:

Docker Compose File

This is the docker-compose.yml file used to run the entire Gitlab on-premise.

version: "3.4"

services:
  gitlabpremise:
    image: gitlab/gitlab-ce:13.9.1-ce.0
    hostname: gitlab.mydomain.com
    ports:
      - "8080:80"
      - "22:22"
      - "5000:5050"
    volumes:
      - /var/data/gitlab/config:/etc/gitlab
      - /var/data/gitlab/logs:/var/log/gitlab
      - /var/data/gitlab/data:/var/opt/gitlab
    restart: always
    environment:
      GITLAB_OMNIBUS_CONFIG: |
        external_url 'https://gitlab.mydomain.com'
        nginx['listen_port'] = 80
        nginx['listen_https'] = false
        user['git_user_email'] = "gitlab@mydomain.com"
        gitlab_rails['gitlab_ssh_host'] = "git.mydomain.com"
        gitlab_rails['gitlab_email_from'] = 'gitlab@mydomain.com'
        gitlab_rails['smtp_enable'] = true
        gitlab_rails['smtp_address'] = "smtp.eu.mailgun.org"
        gitlab_rails['smtp_port'] = 587
        gitlab_rails['smtp_authentication'] = "plain"
        gitlab_rails['smtp_enable_starttls_auto'] = true
        gitlab_rails['smtp_user_name'] = "postmaster@mg.mydomain.com"
        gitlab_rails['smtp_password'] = "mg_password"
        gitlab_rails['smtp_domain'] = "mg.mydomain.com"
        gitlab_rails['registry_enabled'] = false
        gitlab_rails['registry_path'] = "/var/opt/gitlab/gitlab-rails/shared/registry"
        registry_external_url 'https://acr.mydomain.com'
        registry['enable'] = true
        registry['compatibility_schema1_enabled'] = false
        registry_nginx['enable'] = true
        registry_nginx['listen_port'] = 5050
        registry_nginx['listen_https'] = false
        registry_nginx['proxy_set_headers'] = {
          "Host"              => "$$http_host",
          "X-Real-IP"         => "$$remote_addr",
          "X-Forwarded-For"   => "$$proxy_add_x_forwarded_for",
          "X-Forwarded-Proto" => "https",
          "X-Forwarded-Ssl"   => "on"
        }
        registry['env'] = {
          "REGISTRY_HTTP_RELATIVEURLS" => false
        }
        registry['log_level'] = "debug"
        gitlab_rails['gitlab_default_projects_features_container_registry'] = true

Notes

  • The ‘$$’ is used instead of ‘$’ because docker-compose seems to treat $ as environment variables within docker-compose.yml. See more.
  • Both ‘registry_nginx['listen_https'] = false’ and ‘nginx['listen_https'] = false’ because everything is behind a reverse proxy. I however don’t think this is causing the issue because Checklist Item #1 and #2 works just fine even behind reverse proxy providing https. Also, manually going to https://acr.mydomain.com/v2/ does seem to yield reasonable result, it’s just that there was an extra colon : inserted in the end or maybe i’m dealing with a different error here and the extra : is just formatting.
  • I’m expecting that the yml config above will use the internal registry of gitlab on the same docker-compose setup.

Docker Client

➜  ~ docker --version
Docker version 20.10.2, build 2291f61

Docker on Server

➜  ~ docker-compose --version
docker-compose version 1.28.4, build cabd5cfb
➜  ~ docker --version
Docker version 20.10.3, build 48d30b5

Logs

==> /var/log/gitlab/registry/current <==
2021-02-28_12:41:30.03994 {"content_type":"text/plain; charset=utf-8","correlation_id":"01EZMAGX5QY91B4SYEAAN3YNV0","duration_ms":0,"host":"localhost:80","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"127.0.0.1:49820","remote_ip":"127.0.0.1","status":404,"system":"http","time":"2021-02-28T12:41:30Z","ttfb_ms":0,"uri":"http://localhost:80/help","user_agent":"curl/7.74.0-DEV","written_bytes":19}

==> /var/log/gitlab/gitlab-exporter/current <==
2021-02-28_12:41:32.43173 127.0.0.1 - - [28/Feb/2021:12:41:32 UTC] "GET /database HTTP/1.1" 200 1093
2021-02-28_12:41:32.43193 - -> /database

==> /var/log/gitlab/registry/current <==
2021-02-28_12:41:33.15453 time="2021-02-28T12:41:33Z" level=debug msg="authorizing request" correlation_id=01EZMAH070HW4SAEEDWJZYNYSC go_version=go1.14.7 root_repo=
2021-02-28_12:41:33.15468 time="2021-02-28T12:41:33Z" level=warning msg="error authorizing context: authorization token required" correlation_id=01EZMAH070HW4SAEEDWJZYNYSC go_version=go1.14.7 root_repo=
2021-02-28_12:41:33.15469 {"content_type":"application/json","correlation_id":"01EZMAH070HW4SAEEDWJZYNYSC","duration_ms":2,"host":"acr.mydomain.com","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"127.0.0.1:49822","remote_ip":"192.168.1.1","status":401,"system":"http","time":"2021-02-28T12:41:33Z","ttfb_ms":2,"uri":"/v2/","user_agent":"docker/20.10.2 go/go1.13.15 git-commit/8891c58 kernel/4.19.121-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/20.10.2 \\(darwin\\))","written_bytes":87}

==> /var/log/gitlab/nginx/gitlab_registry_access.log <==
192.168.224.1 - - [28/Feb/2021:12:41:33 +0000] "GET /v2/ HTTP/1.1" 401 87 "" "docker/20.10.2 go/go1.13.15 git-commit/8891c58 kernel/4.19.121-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/20.10.2 \x5C(darwin\x5C))" -

==> /var/log/gitlab/gitlab-exporter/current <==
2021-02-28_12:41:34.04159 127.0.0.1 - - [28/Feb/2021:12:41:33 UTC] "GET /sidekiq HTTP/1.1" 200 57997
2021-02-28_12:41:34.04164 - -> /sidekiq
2021-02-28_12:41:38.76751 127.0.0.1 - - [28/Feb/2021:12:41:38 UTC] "GET /ruby HTTP/1.1" 200 996
2021-02-28_12:41:38.76757 - -> /ruby

==> /var/log/gitlab/registry/current <==
2021-02-28_12:41:39.05071 time="2021-02-28T12:41:39Z" level=debug msg="filesystem.Stat(\"/\")" environment=production go_version=go1.14.7 instance_id=11b94e3f-985c-4f8b-a2f3-051742605b1a service=registry trace_duration="55.849µs" trace_file=/var/cache/omnibus/src/registry/src/github.com/docker/distribution/registry/storage/driver/base/base.go trace_func="github.com/docker/distribution/registry/storage/driver/base.(*Base).Stat" trace_id=47efba4d-c70c-454c-bbe5-36903dc4dfad trace_line=155 version=v3.0.0-gitlab

Btw, why is root_repo= blank in the log above? Is that normal? I may have missed something here to.

Relevant Refs

Just to make sure I’m not missing anything, i also tried several permutations of the following:

No reverse proxy and directly exposing IP of the server. Still the same response as before but this time the certificate is separately generated and pasted in shared volume path and not generated by the reverse proxy. This skips reverse proxy altogether.

version: "3.4"

services:
  gitlabpremise:
    image: gitlab/gitlab-ce:13.9.1-ce.0
    hostname: gitlab.mydomain.com
    ports:
      - "8080:80"
      - "22:22"
      - "443:5050"
    volumes:
      - /var/data/gitlab/config:/etc/gitlab
      - /var/data/gitlab/logs:/var/log/gitlab
      - /var/data/gitlab/data:/var/opt/gitlab
    restart: always
    environment:
      GITLAB_OMNIBUS_CONFIG: |
        external_url 'https://gitlab.mydomain.com'
        nginx['listen_port'] = 80
        nginx['listen_https'] = false
        user['git_user_email'] = "gitlab@mydomain.com"
        gitlab_rails['gitlab_ssh_host'] = "git.mydomain.com"
        gitlab_rails['gitlab_email_from'] = 'gitlab@mydomain.com'
        gitlab_rails['smtp_enable'] = true
        gitlab_rails['smtp_address'] = "smtp.eu.mailgun.org"
        gitlab_rails['smtp_port'] = 587
        gitlab_rails['smtp_authentication'] = "plain"
        gitlab_rails['smtp_enable_starttls_auto'] = true
        gitlab_rails['smtp_user_name'] = "postmaster@mg.mydomain.com"
        gitlab_rails['smtp_password'] = "mg_password"
        gitlab_rails['smtp_domain'] = "mg.mydomain.com"
        gitlab_rails['registry_enabled'] = false
        gitlab_rails['registry_path'] = "/var/opt/gitlab/gitlab-rails/shared/registry"
        registry_external_url 'https://acr.mydomain.com'
        registry['enable'] = true
        registry['compatibility_schema1_enabled'] = false
        registry_nginx['enable'] = true
        registry_nginx['listen_port'] = 5050
        registry_nginx['listen_https'] = true
        registry_nginx['ssl_certificate'] = "/etc/gitlab/ssl/acr.mydomain.com.crt"
        registry_nginx['ssl_certificate_key'] = "/etc/gitlab/ssl/acr.mydomain.com.key"
        registry['env'] = {
          "REGISTRY_HTTP_RELATIVEURLS" => true
        }
        registry['log_level'] = "debug"
        gitlab_rails['gitlab_default_projects_features_container_registry'] = true

and Turning REGISTRY_HTTP_RELATIVEURLS, to true and false:

        registry['env'] = {
          "REGISTRY_HTTP_RELATIVEURLS" => true
        }

and

        registry['env'] = {
          "REGISTRY_HTTP_RELATIVEURLS" => false
        }

Whether I’m behind a reverse proxy or directly exposing HTTPS Server IP via DNS, it seems the behaviour is the same.

$ docker login acr.mydomain.com -u my_user -p my_password
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
Error response from daemon: Get https://acr.mydomain.com/v2/: unable to decode token response: invalid character '<' looking for beginning of value

There is no 2FA enabled so the password used to login to Container Registry should be the user password and not the access_token.

Using curl still shows a response on the registry domain:

$ curl https://acr.mydomain.com/v2/
{"errors":[{"code":"UNAUTHORIZED","message":"authentication required","detail":null}]}

And to try my luck, I also tried this:

version: "3.4"

services:
  gitlabpremise:
    image: gitlab/gitlab-ce:13.9.1-ce.0
    hostname: gitlab.mydomain.com
    ports:
      - "8080:80"
      - "22:22"
      - "443:5050"
    volumes:
      - /var/data/gitlab/config:/etc/gitlab
      - /var/data/gitlab/logs:/var/log/gitlab
      - /var/data/gitlab/data:/var/opt/gitlab
    restart: always
    environment:
      GITLAB_OMNIBUS_CONFIG: |
        external_url 'https://gitlab.mydomain.com'
        nginx['listen_port'] = 80
        nginx['listen_https'] = false
        user['git_user_email'] = "gitlab@mydomain.com"
        gitlab_rails['gitlab_ssh_host'] = "git.mydomain.com"
        gitlab_rails['gitlab_email_from'] = 'gitlab@mydomain.com'
        gitlab_rails['smtp_enable'] = true
        gitlab_rails['smtp_address'] = "smtp.eu.mailgun.org"
        gitlab_rails['smtp_port'] = 587
        gitlab_rails['smtp_authentication'] = "plain"
        gitlab_rails['smtp_enable_starttls_auto'] = true
        gitlab_rails['smtp_user_name'] = "postmaster@mg.mydomain.com"
        gitlab_rails['smtp_password'] = "mg_password"
        gitlab_rails['smtp_domain'] = "mg.mydomain.com"
        gitlab_rails['registry_enabled'] = true
        gitlab_rails['registry_host'] = "acr.mydomain.com"
        gitlab_rails['registry_port'] = "443"
        gitlab_rails['registry_path'] = "/var/opt/gitlab/gitlab-rails/shared/registry"
        gitlab_rails['registry_api_url'] = "https://acr.mydomain.com"
        gitlab_rails['registry_key_path'] = "/var/opt/gitlab/gitlab-rails/certificate.key"
        gitlab_rails['registry_issuer'] = "omnibus-gitlab-issuer"
        registry_external_url 'https://acr.mydomain.com'
        registry_nginx['enable'] = true
        registry_nginx['listen_port'] = "5050"
        registry_nginx['listen_https'] = true
        registry_nginx['ssl_certificate'] = "/etc/gitlab/ssl/acr.mydomain.com.crt"
        registry_nginx['ssl_certificate_key'] = "/etc/gitlab/ssl/acr.mydomain.com.key"
        registry['enable'] = true
        registry['compatibility_schema1_enabled'] = false
        registry['env'] = {
          "REGISTRY_HTTP_RELATIVEURLS" => true
        }
        registry['log_level'] = "debug"
        gitlab_rails['gitlab_default_projects_features_container_registry'] = true

But I have a feeling gitlab_rails['registry_enabled'] = true should be false on my end because I intend to use the built in registry. I have a feeling this is only set to true if there is an external registry being used.

This results in the same exact error as before.

$ docker login acr.mydomain.com -u my_username -p my_password
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
Error response from daemon: Get https://acr.mydomain.com/v2/: unable to decode token response: invalid character '<' looking for beginning of value

I got a bit confused where’s the proper channel to request for support on this kind of things but I’ll also update this posting here whenever I get updates from any of the channels I have posted this question on.

1 Like