Gitlab Runner: x509: certificate signed by unknown authority

Hi,

you mean the login to the registry in the before_script section? AFAIK this always has been the case, that’s also inside the Docker .gitlab-ci.yml template (example here: https://docs.gitlab.com/ee/ci/docker/using_docker_build.html#container-registry-examples)

One thing I could also think of - the DOCKER_TLS_CERTDIR variable. This one is new since a few versions since Docker hardened things with TLS.

I’m not sure if that has an impact though.

Cheers,
Michael

@dnsmichi This is not only about the gitlab runners. I get the same error also when I try to log into the registry with docker (command as mentioned before)

Ok, then back to the permissions. I do believe that the private key is not readable by the gitlab user git.

Same as pointed out here: https://codereviewvideos.com/blog/fixed-error-authorizing-context-authorization-token-required/

Is there a user git in a dockerized gitlab?

@dnsmichi My current permissions for /etc/gitlab:

-rw------- 1 root root 15567 Dec 10 08:55 gitlab-secrets.json
-rw------- 1 root root 89490 Dec 10 07:57 gitlab.rb
-rw------- 1 root root   227 Sep  9 17:40 ssh_host_ecdsa_key
-rw-r--r-- 1 root root   179 Sep  9 17:40 ssh_host_ecdsa_key.pub
-rw------- 1 root root   411 Sep  9 17:40 ssh_host_ed25519_key
-rw-r--r-- 1 root root    99 Sep  9 17:40 ssh_host_ed25519_key.pub
-rw------- 1 root root  1675 Sep  9 17:40 ssh_host_rsa_key
-rw-r--r-- 1 root root   399 Sep  9 17:40 ssh_host_rsa_key.pub
drwxr-xr-x 3 root root  4096 Dec  9 19:10 ssl/
drwxr-xr-x 2 root root  4096 Sep  9 17:40 trusted-certs/

No, sorry. I did not try that out, I’m used to running GitLab directly via package installation.

Looks the same now in Docker here:

root@gitlab:/# ls -la /etc/gitlab/
total 140
drwxrwxr-x 11 root root   352 Dec 14 21:49 .
drwxr-xr-x  1 root root  4096 Dec 14 21:48 ..
-rw-------  1 root root 15447 Dec 14 21:49 gitlab-secrets.json
-rw-------  1 root root 97928 Dec 14 21:48 gitlab.rb
-rw-------  1 root root   227 Dec 14 21:48 ssh_host_ecdsa_key
-rw-r--r--  1 root root   173 Dec 14 21:48 ssh_host_ecdsa_key.pub
-rw-------  1 root root   399 Dec 14 21:48 ssh_host_ed25519_key
-rw-r--r--  1 root root    93 Dec 14 21:48 ssh_host_ed25519_key.pub
-rw-------  1 root root  1675 Dec 14 21:48 ssh_host_rsa_key
-rw-r--r--  1 root root   393 Dec 14 21:48 ssh_host_rsa_key.pub
drwxr-xr-x  2 root root    64 Dec 14 21:49 trusted-certs

The thing is that you’re having the ssl directory where the actual key file is located. It might be the case that permissions are wrong in there. Please show a full ls -lahR /etc/gitlab.

@dnsmichi Sorry for the late reply. Here are the whole permissions for /etc/gitlab:

root@2357c223d862:/# ls -lahR /etc/gitlab/
/etc/gitlab/:
total 148K
drwxrwxr-x 4 root root 4.0K Dec  9 19:11 .
drwxr-xr-x 1 root root 4.0K Dec 10 07:47 ..
-rw------- 1 root root  16K Dec 10 08:55 gitlab-secrets.json
-rw------- 1 root root  88K Dec 10 07:57 gitlab.rb
-rw------- 1 root root  227 Sep  9 17:40 ssh_host_ecdsa_key
-rw-r--r-- 1 root root  179 Sep  9 17:40 ssh_host_ecdsa_key.pub
-rw------- 1 root root  411 Sep  9 17:40 ssh_host_ed25519_key
-rw-r--r-- 1 root root   99 Sep  9 17:40 ssh_host_ed25519_key.pub
-rw------- 1 root root 1.7K Sep  9 17:40 ssh_host_rsa_key
-rw-r--r-- 1 root root  399 Sep  9 17:40 ssh_host_rsa_key.pub
drwxr-xr-x 3 root root 4.0K Dec  9 19:10 ssl
drwxr-xr-x 2 root root 4.0K Sep  9 17:40 trusted-certs

/etc/gitlab/ssl:
total 12K
drwxr-xr-x 3 root root 4.0K Dec  9 19:10 .
drwxrwxr-x 4 root root 4.0K Dec  9 19:11 ..
drwxr-x--- 2 root root 4.0K Dec  9 19:10 mydomain

/etc/gitlab/ssl/mydomain:
total 16K
drwxr-x--- 2 root root 4.0K Dec  9 19:10 .
drwxr-xr-x 3 root root 4.0K Dec  9 19:10 ..
-rw-r----- 1 root root 3.5K Dec  9 19:10 fullchain.pem
-rw-r----- 1 root root 1.7K Dec  9 19:10 privkey.pem

/etc/gitlab/trusted-certs:
total 8.0K
drwxr-xr-x 2 root root 4.0K Sep  9 17:40 .
drwxrwxr-x 4 root root 4.0K Dec  9 19:11 ..

Hi,

no worries, everyone got work to do. I really appreciate users like you who follow advice, provide details and do not stress it. I also know that differently from open source users who demand enterprise support for free.

This is a blind guess, but I’d try:

chmod 600 /etc/gitlab/ssl/mydomain/privkey.pem
chmod 644 /etc/gitlab/ssl/mydomain/fullchain.pem

Then restart … well all services which load these certificates.

If that still doesn’t work, I’d go with one route - either the http proxy up front, or full mapped ports.

Cheers,
Michael

Hi,

I also really appreciate your help!

I’ve set the permissions like you said:

/etc/gitlab/ssl/mydomain:
total 16K
drwxr-x--- 2 root root 4.0K Dec  9 19:10 .
drwxr-xr-x 3 root root 4.0K Dec  9 19:10 ..
-rw------- 1 root root 3.5K Dec  9 19:10 fullchain.pem
-rw------- 1 root root 1.7K Dec  9 19:10 privkey.pem

But sadly the issue is not fixed yet. Just to summarise everything:

• I cannot login to docker registry. There is no difference between trying with a gitlab runner or trying to log in with the command docker login coding.mydomain:5005.
• I get the following error when I try to log in: Error response from daemon: Get https://coding.mydomain:5005/v2/: denied: access forbidden
• What have I tried to log in?: I tried it with my user password and also generated an Access Token with all permissions.
• What have I done before that it stopped working?: I moved gitlab from mydomain to coding.mydomain by changing the external_url variable and restarting gitlab. Also the ports 80 and 443 of gitlab are running behind a reverse proxy with the same ssl certificate as the docker registry now. The docker registry port is still mapped directly to the gitlab container.
• The error message of a docker login coding.mydomain:5005 is attached at the end of this reply.

Already fixed:

• SSL problems: I don’t understand why I can’t change the file extension from a pem formatted key. This has to be a gitlab thing but now SSL is working correctly.

My next questions:

• Reverse Proxy: Is the api request going over the Reverse Proxy or is the API for the docker registry listening on localhost?
• Which other things can I try to find the error?
• Can I reset the registry somehow? Maybe it is corrupt or something. I don’t need the images which were uploaded to the registry before.

The error message from docker login coding.mydomain:5005:

==> /var/log/gitlab/registry/current <==
2019-12-19_09:42:50.35547 time="2019-12-19T09:42:50.35513125Z" level=warning msg="error authorizing context: authorization token required" go.version=go1.12.13 http.request.host="coding.mydomain:5005" http.request.id=fa3d1849-7eec-48b2-a940-35dd454d6aed http.request.method=GET http.request.remoteaddr=myip http.request.uri="/v2/" http.request.useragent="docker/18.09.7 go/go1.10.1 git-commit/2d0083d kernel/4.15.0-72-generic os/linux arch/amd64 UpstreamClient(Docker-Client/18.09.7 \(linux\))"
2019-12-19_09:42:50.35553 127.0.0.1 - - [19/Dec/2019:09:42:50 +0000] "GET /v2/ HTTP/1.1" 401 87 "" "docker/18.09.7 go/go1.10.1 git-commit/2d0083d kernel/4.15.0-72-generic os/linux arch/amd64 UpstreamClient(Docker-Client/18.09.7 \\(linux\\))"

Hi,

I forgot to ask about your .gitlab-ci.yml config file during this analysis, since we were only looking at connection troubleshooting before. Maybe there is something wrong the job config, please share it with us.

Cheers,
Michael

Hi @dnsmichi,

I don’t know why it should stop working when I only changed the domain in the ci file and nothing else. Also a simple docker login on the command line is not working. But this is who I pull the image in gitlab-ci.yml:

image: coding.mydomain:5005/project_group/project_name/ansible:latest

Hi,

I was also asking for the full config file with the intention of re-creating your environment here in a test setup over the holidays (or maybe January). The more common things we have the better. I truly don’t understand yet why the Docker login fails with the granted token.

Cheers,
Michael

Which config do you mean? The gitlab.rb config file, the gitlab runner or the gitlab-ci.yml?

This is why I don’t think that it is a error in my gitlab-ci.yml file.

I was asking for the config file to create a 1:1 reproducer here. If you can think of other ways to reproduce this, please share them. Right now I do consider a problem with the mix of Docker containers and native port mapping. I bet that this problem doesn’t exist if you run GitLab outside of the container, or remove the Nginx proxy up front. Somehow the packets are going a weird route, or something else is intercepting the communication.

Hi @dnsmichi,

sorry for not replying but I was really busy the last weeks.
Thanks for your support!
This is my configuration of my gitlab setup:

My Nginx config for the WebGUI:

# HTTP server configuration
server {
    if ($host = coding.kapra.de) {
        return 301 https://$host$request_uri;
    } # managed by Certbot


        # HTTP configuration
        listen <MYIP>:80;
        listen [::]:80;

        server_name coding.mydomain.de;

        return 301 https://$host$request_uri;
}

server {
        # SSL configuration
        listen MYIP:443 ssl;
        listen [::]:443 ssl;

        server_name coding.mydomain.de;

        ssl on;
    ssl_certificate /etc/letsencrypt/live/coding.MYDOMAIN.de/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/coding.MYDOMAIN.de/privkey.pem; # managed by Certbot
        ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
        ssl_ciphers HIGH:!aNULL:!MD5;

        location / {
            proxy_pass http://127.0.0.1:8083/;
            proxy_set_header Host $http_host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
            client_max_body_size 0;
        }
}

My docker environment:

version: '2'

services:
  gitlab:
    restart: always
    image: gitlab/gitlab-ce
    container_name: gitlab_compose
    ports:
    - "127.0.0.1:8083:80"
    - "0.0.0.0:2222:22"
    - "0.0.0.0:5005:5005"
    volumes:
    - /etc/gitlab:/etc/gitlab
    - /var/opt/gitlab:/var/opt/gitlab
    - /var/log/gitlab:/var/log/gitlab

This is my gitlab-config (I deleted everything what is commented out):

## GitLab URL

##! URL on which GitLab will be reachable.

##! For more details on configuring external_url see:

##! https://docs.gitlab.com/omnibus/settings/configuration.html#configuring-the-external-url-for-gitlab

external_url 'http://coding.mydomain.de'

## Roles for multi-instance GitLab

##! The default is to have no roles enabled, which results in GitLab running as an all-in-one instance.

##! Options:

##! redis_sentinel_role redis_master_role redis_slave_role geo_primary_role geo_secondary_role

##! For more details on each role, see:

##! https://docs.gitlab.com/omnibus/roles/README.html#roles

##!

# roles ['redis_sentinel_role', 'redis_master_role']

## Legend

##! The following notations at the beginning of each line may be used to

##! differentiate between components of this file and to easily select them using

##! a regex.

##! ## Titles, subtitles etc

##! ##! More information - Description, Docs, Links, Issues etc.

##! Configuration settings have a single # followed by a single space at the

##! beginning; Remove them to enable the setting.

##! **Configuration settings below are optional.**

##! **The values currently assigned are only examples and ARE NOT the default

##! values.**

################################################################################

################################################################################

## Configuration Settings for GitLab CE and EE ##

################################################################################

################################################################################

################################################################################

## gitlab.yml configuration

##! Docs: https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/doc/settings/gitlab.yml.md

################################################################################

# gitlab_rails['gitlab_ssh_host'] = 'mydomain.de:2222'

# gitlab_rails['time_zone'] = 'UTC'

### Email Settings

gitlab_rails['gitlab_email_enabled'] = true

gitlab_rails['gitlab_email_from'] = 'gitlab@mydomain.de'

gitlab_rails['gitlab_email_display_name'] = 'MyDomain Gitlab'

gitlab_rails['gitlab_email_reply_to'] = 'gitlab@mydomain.de'

gitlab_rails['gitlab_email_subject_suffix'] = ''

### GitLab Shell settings for GitLab

gitlab_rails['gitlab_shell_ssh_port'] = 2222

# SMTP

gitlab_rails['smtp_enable'] = true

gitlab_rails['smtp_address'] = "mail.mydomain.de"

gitlab_rails['smtp_port'] = 587

gitlab_rails['smtp_user_name'] = "gitlab@mydomain.de"

gitlab_rails['smtp_password'] = "<Password>"

gitlab_rails['smtp_domain'] = "mail.mydomain.de"

gitlab_rails['smtp_authentication'] = "login"

gitlab_rails['smtp_enable_starttls_auto'] = true

gitlab_rails['smtp_tls'] = false

gitlab_rails['smtp_openssl_verify_mode'] = 'none'

# If your SMTP server does not like the default 'From: gitlab@localhost' you

# can change the 'From' with this setting.

gitlab_rails['gitlab_email_from'] = 'gitlab@mydomain.de'

gitlab_rails['gitlab_email_reply_to'] = 'gitlab@mydomain.de'

################################################################################

## Container Registry settings

##! Docs: https://docs.gitlab.com/ce/administration/container_registry.html

################################################################################

registry_external_url 'https://coding.mydomain.de:5005'

### Settings used by GitLab application

gitlab_rails['registry_enabled'] = true

# gitlab_rails['registry_host'] = "coding.mydomain.de"

# gitlab_rails['registry_port'] = "5005"

# gitlab_rails['registry_path'] = "/var/opt/gitlab/gitlab-rails/shared/registry"

################################################################################

## Registry NGINX

################################################################################

# All the settings defined in the "GitLab Nginx" section are also available in

# this "Registry NGINX" section, using the key `registry_nginx`. However, those

# settings should be explicitly set. That is, settings given as

# `nginx['some_setting']` WILL NOT be automatically replicated as

# `registry_nginx['some_setting']` and should be set separately.

# Below you can find settings that are exclusive to "Registry NGINX"

registry_nginx['enable'] = true

registry_nginx['ssl_certificate'] = "/etc/gitlab/ssl/coding.mydomain.de/fullchain.pem"

registry_nginx['ssl_certificate_key'] = "/etc/gitlab/ssl/coding.mydomain.de/privkey.pem"

# registry_nginx['proxy_set_headers'] = {

# "Host" => "$http_host",

# "X-Real-IP" => "$remote_addr",

# "X-Forwarded-For" => "$proxy_add_x_forwarded_for",

# "X-Forwarded-Proto" => "https",

# "X-Forwarded-Ssl" => "on"

# }

################################################################################

# Let's Encrypt integration

################################################################################

letsencrypt['enable'] = false

# letsencrypt['contact_emails'] = [] # This should be an array of email addresses to add as contacts

# letsencrypt['group'] = 'root'

# letsencrypt['key_size'] = 2048

# letsencrypt['owner'] = 'root'

# letsencrypt['wwwroot'] = '/var/opt/gitlab/nginx/www'

# See http://docs.gitlab.com/omnibus/settings/ssl.html#automatic-renewal for more on these sesttings

# letsencrypt['auto_renew'] = true

# letsencrypt['auto_renew_hour'] = 0

# letsencrypt['auto_renew_minute'] = nil # Should be a number or cron expression, if specified.

# letsencrypt['auto_renew_day_of_month'] = "*/4"

Hi,

No worries, I bet I was even busier

Also, a fresh look on things does not hurt. Lately I had been playing with Nginx/Docker proxying myself while building Ghost for https://dnsmichi.at - maybe we are running in proxy header problems here.

These settings sound interesting, how about enabling them and test again?

# registry_nginx['proxy_set_headers'] = {
# "Host" => "$http_host",
# "X-Real-IP" => "$remote_addr",
# "X-Forwarded-For" => "$proxy_add_x_forwarded_for",
# "X-Forwarded-Proto" => "https",
# "X-Forwarded-Ssl" => "on"
# }

IIRC the registry itself runs inside the container hidden behind an Nginx proxy too. Partially inspired by Registry | Docker Docs

Cheers,
Michael

Hi @dnsmichi,

I think my reverse proxy is not configured correctly.
https://gitlab.com/gitlab-org/gitlab-recipes/blob/master/web-server/nginx/gitlab-omnibus-ssl-nginx.conf

This is what I found and tried but it is not working. I also don’t really understand how to integrate it into a docker environment.

Hi,

I thought we are on the level of

• GitLab Web via Docker + Nginx proxy up front
• Docker registry in Docker, direct port mapping to 5005

? Therefore the Docker registry settings are important, and the configuration inside the container. Which is why I would look into the registry_nginx settings.

Cheers,
Michael

Yes you are right. I was thought in the wrong way there.
Uncommenting the option you mentioned above does not
solve the problem.

Ok, another idea - what happens if you add port 5005 to your Nginx proxy, similar to 80 and 443 and their settings? This would be the logical termination point for TLS then.

Cheers,
Michael