Cannot configure docker registry on separate vm

Good day. I have two VMs with ubuntu 18.04. One for gitlab omnibus latest, and another one for registry. I want to make this vms working together.
I also have 2 certificates: one - wildcard for my domain, gitlab and docker are using it (tempgitlab.domain.com, registry.domain.com). And another keypair i made for connecting:

openssl req -nodes -newkey rsa:4096 -keyout registry-auth.key -out registry-auth.csr -subj “/CN=gitlab-issuer”
openssl x509 -in registry-auth.csr -out registry-auth.crt -req -signkey registry-auth.key -days 3650

I run registry with this command

docker run -d --restart=always -p 443:443 --name registry -v /home/ubuntu/certs:/certs -v /etc/gitlab/registry-certs:/etc/gitlab/registry-certs -e REGISTRY_AUTH_TOKEN_REALM=https://tempgitlab.domain.com/jwt/auth -e REGISTRY_AUTH_TOKEN_SERVICE=container_registry -e REGISTRY_AUTH_TOKEN_ISSUER=gitlab-issuer -e REGISTRY_AUTH_TOKEN_ROOTCERTBUNDLE=/etc/gitlab/registry-certs/registry-auth.crt -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/domain.crt -e REGISTRY_HTTP_TLS_KEY=/certs/domain.key --name docker-registry registry:2

My gitlab config (registry part):

registry_external_url ‘https://registry.domain.com/
gitlab_rails[‘registry_enabled’] = true
gitlab_rails[‘registry_host’] = “registry.domain.com
gitlab_rails[‘registry_port’] = “443”
registry[‘internal_key’] = “-----BEGIN RSA PRIVATE KEY-----\nMi…”
gitlab_rails[‘registry_api_url’] = “https://registry.domain.com:443
gitlab_rails[‘registry_key_path’] = “/etc/gitlab/registry-certs/registry-auth.key”
gitlab_rails[‘registry_issuer’] = “gitlab-issuer”

And with this settings docker runs perfectly. I can login with my gitlab creds, i can push in repository only if i have permissions. Awesome. But while docker cli runs good, i have an error 500, when i’m trying to access registry page on gitlab.

LOG from docker

2018/12/22 10:35:26 http: TLS handshake error from 192.168.78.131:43818: remote error: tls: unknown certificate authority

LOG from gitlab

Completed 500 Internal Server Error in 58ms (ActiveRecord: 3.1ms | Elasticsearch: 0.0ms)

Faraday::SSLError (SSL_connect returned=1 errno=0 state=error: certificate verify failed):
lib/container_registry/client.rb:21:in repository_tags' app/models/container_repository.rb:38:in manifest’
app/models/container_repository.rb:43:in tags' app/models/container_repository.rb:55:in has_tags?’
lib/gitlab/metrics/instrumentation.rb:159:in block in has_tags?' lib/gitlab/metrics/method_call.rb:34:in measure’
lib/gitlab/metrics/instrumentation.rb:159:in has_tags?' app/controllers/projects/registry/repositories_controller.rb:46:in block (2 levels) in ensure_root_container_repository!’
app/controllers/projects/registry/repositories_controller.rb:45:in tap' app/controllers/projects/registry/repositories_controller.rb:45:in block in ensure_root_container_repository!’
app/controllers/projects/registry/repositories_controller.rb:42:in tap' app/controllers/projects/registry/repositories_controller.rb:42:in ensure_root_container_repository!’
lib/gitlab/i18n.rb:55:in with_locale' lib/gitlab/i18n.rb:61:in with_user_locale’
app/controllers/application_controller.rb:427:in set_locale' lib/gitlab/middleware/multipart.rb:101:in call’
lib/gitlab/request_profiler/middleware.rb:14:in call' ee/lib/gitlab/jira/middleware.rb:15:in call’
lib/gitlab/middleware/go.rb:17:in call' lib/gitlab/etag_caching/middleware.rb:11:in call’
lib/gitlab/middleware/rails_queue_duration.rb:22:in call' lib/gitlab/metrics/rack_middleware.rb:15:in block in call’
lib/gitlab/metrics/transaction.rb:53:in run' lib/gitlab/metrics/rack_middleware.rb:15:in call’
lib/gitlab/middleware/read_only/controller.rb:40:in call' lib/gitlab/middleware/read_only.rb:16:in call’
lib/gitlab/middleware/basic_health_check.rb:25:in call' lib/gitlab/request_context.rb:20:in call’
lib/gitlab/metrics/requests_rack_middleware.rb:27:in call' lib/gitlab/middleware/release_env.rb:10:in call’

I’m going crazy with all of this certificates and hope somebody could help me.

I’ve also done on gitlab

cat domain.key > domain.pem && cat domain.crt >> domain.pem
cp domain.pem /etc/gitlab/trusted-certs

didn’t help :slight_smile:

and maybe this can help:

openssl s_client -connect registry.domain.com:443 CONNECTED(00000003) depth=0 OU = Domain Control Validated, OU = PositiveSSL Wildcard, CN = *.domain.com verify error:num=20:unable to get local issuer certificate verify return:1 depth=0 OU = Domain Control Validated, OU = PositiveSSL Wildcard, CN = *.domain.com verify error:num=21:unable to verify the first certificate verify return:1

Oh, It was so simple. my wildcard certificate (registry-auth.crt) for docker had to include intermediate and root certificate… :slight_smile:

May you please share instruction how to cope it step by step?