Unable to login to registry with two separate domains

Infrastructure as Code & Cloud Native
1

Hello, I have a problem with my GitLab install. This morning I enabled registry on a separate domain because I can’t use ports with Cloudflare (and also because I find that more “beautiful”… :smiley: ). But now, I can’t connect on my registry with the docker login registry.gitlab.domain.tld command. I have this error message:

Error response from daemon: Get "https://registry.gitlab.domain.tld/v2/": denied: access forbidden

I configured my instance behind a reverse proxy with Apache2. Here is my virtualhost config file

<VirtualHost *:80>
  ServerName gitlab.domain.tld
  ServerSignature Off

  RewriteEngine on
  RewriteCond %{HTTPS} !=on
  RewriteRule .* https://%{SERVER_NAME}%{REQUEST_URI} [NE,R,L]
</VirtualHost>

<VirtualHost *:443>
  SSLEngine on
    SSLCertificateFile /etc/letsencrypt/live/gitlab.domain.tld/cert.pem
    SSLCertificateKeyFile /etc/letsencrypt/live/gitlab.domain.tld/privkey.pem
    SSLCertificateChainFile /etc/letsencrypt/live/gitlab.domain.tld/chain.pem
    SSLProtocol all -SSLv2 -SSLv3
    SSLHonorCipherOrder on
    SSLCompression off
    SSLOptions +StrictRequire
    SSLCipherSuite ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA
    Header always set Strict-Transport-Security "max-age=31536000; includeSubDomains"

  ServerName gitlab.domain.tld
  ServerSignature Off

  ProxyPreserveHost On

  # Ensure that encoded slashes are not decoded but left in their encoded state.
  # http://doc.gitlab.com/ce/api/projects.html#get-single-project
  AllowEncodedSlashes NoDecode

  <Location />
    # New authorization commands for apache 2.4 and up
    # http://httpd.apache.org/docs/2.4/upgrading.html#access
    Require all granted

    #Allow forwarding to gitlab-workhorse
    ProxyPassReverse http://127.0.0.1:8180
    ProxyPassReverse http://gitlab.domain.tld
  </Location>

  # Apache equivalent of nginx try files
  # http://serverfault.com/questions/290784/what-is-apaches-equivalent-of-nginxs-try-files
  # http://stackoverflow.com/questions/10954516/apache2-proxypass-for-rails-app-gitlab
  RewriteEngine on

  ########
  ## LIS SOFIANE! 
  ##
  ## Il faut regarder /etc/gitlab/gitlab.rb pour voir les ports utilisés
  ## Check aussi ce post si jamais les images disparaissent à nouveau: https://forum.gitlab.com/t/uploaded-images-dont-shows-up-with-custom-width/64621/3?u=sofianelasri
  ########

  # Don't escape encoded characters in API requests. This makes the IDE work.
  # Don't escape encoded characters in -/refs/*/logs_tree/ or -/tree/, since 12.85,
  # these are pre-encoded, and excluding them causes double-encoding.
  RewriteCond %{REQUEST_URI} ^/api/v\d+/.* [OR]
  RewriteCond %{REQUEST_URI} .*-/branches/.* [OR]
  RewriteCond %{REQUEST_URI} .*-/refs/.*/logs_tree/.* [OR]
  RewriteCond %{REQUEST_URI} .*-/tree/.*
  RewriteRule .* http://127.0.0.1:8182%{REQUEST_URI} [P,QSA,NE]

  # Forward all requests to gitlab-workhorse except existing files like error documents
  RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_FILENAME} !-f [OR]
  RewriteCond %{REQUEST_URI} ^/uploads/.*
  RewriteRule .* http://127.0.0.1:8182%{REQUEST_URI} [P,QSA]

  RequestHeader set X_FORWARDED_PROTO 'https'
  RequestHeader set X-Forwarded-Ssl on

  # needed for downloading attachments
  DocumentRoot /opt/gitlab/embedded/service/gitlab-rails/public

  #Set up apache error documents, if back end goes down (i.e. 503 error) then a maintenance/deploy page is thrown up.
  ErrorDocument 404 /404.html
  ErrorDocument 422 /422.html
  ErrorDocument 500 /500.html
  ErrorDocument 502 /502.html
  ErrorDocument 503 /503.html

  # It is assumed that the log directory is in /var/log/httpd.
  # For Debian distributions you might want to change this to
  # /var/log/apache2.
  LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b" common_forwarded
  ErrorLog ${APACHE_LOG_DIR}/gitlab.domain.tld_error.log
  CustomLog ${APACHE_LOG_DIR}/gitlab.domain.tld_forwarded.log common_forwarded
  CustomLog ${APACHE_LOG_DIR}/gitlab.domain.tld_access.log combined env=!dontlog
  CustomLog ${APACHE_LOG_DIR}/gitlab.domain.tld.log combined

</VirtualHost>

<VirtualHost *:80>
  ServerName registry.gitlab.domain.tld
  ServerSignature Off

  RewriteEngine on
  RewriteCond %{HTTPS} !=on
  RewriteRule .* https://%{SERVER_NAME}%{REQUEST_URI} [NE,R,L]
</VirtualHost>

<VirtualHost *:443>
  SSLEngine on
    SSLCertificateFile /etc/letsencrypt/live/registry.gitlab.domain.tld/cert.pem
    SSLCertificateKeyFile /etc/letsencrypt/live/registry.gitlab.domain.tld/privkey.pem
    SSLCertificateChainFile /etc/letsencrypt/live/registry.gitlab.domain.tld/chain.pem
    SSLProtocol all -SSLv2 -SSLv3
    SSLHonorCipherOrder on
    SSLCompression off
    SSLOptions +StrictRequire
    SSLCipherSuite ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA
    Header always set Strict-Transport-Security "max-age=31536000; includeSubDomains"

  ServerName registry.gitlab.domain.tld
  ServerSignature Off

  ProxyPreserveHost On

  # Ensure that encoded slashes are not decoded but left in their encoded state.
  # http://doc.gitlab.com/ce/api/projects.html#get-single-project
  AllowEncodedSlashes NoDecode

  <Location />
    Require all granted

    ProxyPass http://127.0.0.1:5000/ timeout=900
    ProxyPassReverse http://127.0.0.1:5000/
  </Location>

  Header always set Docker-Distribution-Api-Version "registry/2.0"
  RequestHeader set X-Forwarded-Proto "https"

  # It is assumed that the log directory is in /var/log/httpd.
  # For Debian distributions you might want to change this to
  # /var/log/apache2.
  LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b" common_forwarded
  ErrorLog ${APACHE_LOG_DIR}/registry.gitlab.domain.tld_error.log
  CustomLog ${APACHE_LOG_DIR}/registry.gitlab.domain.tld_forwarded.log common_forwarded
  CustomLog ${APACHE_LOG_DIR}/registry.gitlab.domain.tld_access.log combined env=!dontlog
  CustomLog ${APACHE_LOG_DIR}/registry.gitlab.domain.tld.log combined
</VirtualHost>

I can access to the registry domain without difficulties, it shows me some json code that ask me to authenticate. So I don’t think it’s a problem from my virtualhost configuration. I found some topics that discuss about this problem, here are two of them:

But none of the ones I found helped me. I checked the yaml config file located in /var/opt/gitlab/registry, and I use the right auth.token.realm url with https protocol.

Here is my registry configuration in the gitlab.rb config file:

################################################################################
## Container Registry settings
##! Docs: https://docs.gitlab.com/ee/administration/container_registry.html
################################################################################

registry_external_url 'https://registry.gitlab.domain.tld'

### Settings used by GitLab application
gitlab_rails['registry_enabled'] = true
gitlab_rails['registry_host'] = "registry.gitlab.domain.tld"
# gitlab_rails['registry_port'] = "5005"
# gitlab_rails['registry_path'] = "/var/opt/gitlab/gitlab-rails/shared/registry"

I have no idea how to fix that. :frowning:
I am using GitLab CE v16.1.2.

Thanks in advance for help !

3

Hi @SofianeLasri
anything useful in /var/log/gitlab/registry why it’s denied? Do you see the requests coming to your custom Reverse Proxy and 403 is returning from the upstream?

4

Yes, I have two log lines in the current log file.
Here is the first:

2023-07-24_16:54:41.42940 time="2023-07-24T18:54:41.429+02:00" level=info msg="router info" config_http_addr="127.0.0.1:5000" config_http_host= config_http_net= config_http_prefix= config_http_relative_urls=false correlation_id=01H64D64GNFG0TATW7XJJ573K3 go_version=go1.19.8 method=GET path=/v2/ root_repo= router=gorilla/mux version=v3.76.0-gitlab

The second is a json data that tells me that the client ended in a 401 error:

{
  "content_type": "application/json",
  "correlation_id": "01H64D64GNFG0TATW7XJJ573K3",
  "duration_ms": 0,
  "host": "registry.gitlab.domain.tld",
  "level": "info",
  "method": "GET",
  "msg": "access",
  "proto": "HTTP/1.1",
  "referrer": "",
  "remote_addr": "127.0.0.1:43692",
  "remote_ip": "myIp",
  "status": 401,
  "system": "http",
  "time": "2023-07-24T18:54:41.429+02:00",
  "ttfb_ms": 0,
  "uri": "/v2/",
  "user_agent": "docker/24.0.2 go/go1.20.4 git-commit/659604f kernel/5.15.90.1-microsoft-standard-WSL2 os/linux arch/amd64 UpstreamClient(Docker-Client/24.0.2 \\(windows\\))",
  "written_bytes": 87
}
5

Oh what a shame… I just found why it didn’t worked… I searched all the day why and, the solution was just a simple permission to grant… :skull:
But I don’t understand something. Why did it worked with this token when the registry was configured on the main subdomain ? That’s strange…

The api permission is needed in order use the registry. :man_facepalming: