Introduction to problem
Hi, i deployed Gitlab on Rocky Linux 8 VM and Gitlab-runner on RHEL 9 VM and custom self-signed CA.
Gitlab is up and running and jobs can be picked up with the runners successfully.
I want to create a custom docker image and push it to my container registry that is hosted in Gitlab (not external registry).
Dockerfile in Gitlab.
.gitlab-ci.yml
build image:
image: docker:20.10.17
cache: []
services:
- name: docker:20.10.17-dind
script:
- echo $CI_REGISTRY_PASSWORD | docker login -u $CI_REGISTRY_USER $CI_REGISTRY --password-stdin
- docker build -t $CI_REGISTRY_IMAGE .
- echo $GITLAB_REGISTRY_API_URL
- echo $CI_REGISTRY_IMAGE
- docker push $CI_REGISTRY_IMAGE
Job fails
When the job gets triggered with the above .gitlab-ci.yml file then it fails by retrying to push the image and at the end with the error received unexpected HTTP status: 200 OK
Output of some relevant information/configuration to help debug or reproduce the problem
on Server hosting Gitlab (Rocky 8)
/etc/gitlab/gitlab.rb
[root@magnaslab nginx]# grep -v '^#\|^$' /etc/gitlab/gitlab.rb
external_url 'https://gitlab.magnas.com'
gitlab_rails['gitlab_default_theme'] = 1
gitlab_rails['ldap_enabled'] = true
gitlab_rails['prevent_ldap_sign_in'] = false
gitlab_rails['ldap_servers'] = YAML.load <<-'EOS'
main: # 'main' is the GitLab 'provider ID' of this LDAP server
label: 'magnas'
host: 'DC01.magnas.com'
port: 389
uid: 'sAMAccountName'
encryption: 'plain' # "start_tls" or "simple_tls" or "plain"
active_directory: true
allow_username_or_email_login: true
lowercase_usernames: true
base: ''
group_base: ''
admin_group: ''
sync_ssh_keys: false
EOS
registry_external_url 'https://gitlab.magnas.com'
registry['env'] = {
"REGISTRY_HTTP_RELATIVEURLS" => true
}
registry_nginx['ssl_certificate'] = "/etc/gitlab/ssl/gitlab.magnas.com.crt"
registry_nginx['ssl_certificate_key'] = "/etc/gitlab/ssl/gitlab.magnas.com.key"
gitlab_rails['registry_enabled'] = true
gitlab_rails['registry_host'] = "gitlab.magnas.com"
gitlab_rails['registry_port'] = "80"
registry['enable'] = true
registry['username'] = "registry"
registry['env'] = {
'SSL_CERT_DIR' => "/opt/gitlab/embedded/ssl/certs/"
}
registry['autoredirect'] = true
nginx['enable'] = true
nginx['client_max_body_size'] = '250m'
nginx['redirect_http_to_https'] = true
nginx['redirect_http_to_https_port'] = 80
nginx['ssl_client_certificate'] = "/etc/gitlab/ssl/gitlab.magnas.com.crt"
nginx['ssl_certificate'] = "/etc/gitlab/ssl/gitlab.magnas.com.crt"
nginx['ssl_certificate_key'] = "/etc/gitlab/ssl/gitlab.magnas.com.key"
nginx['listen_addresses'] = ["0.0.0.0", "[::]"]
nginx['listen_port'] = 443
registry_nginx['enable'] = true
registry_nginx['listen_https'] = true
registry_nginx['redirect_http_to_https'] = true
letsencrypt['enable'] = false # GitLab 10.5 and 10.6 require this option
external_url "https://gitlab.magnas.com" # Must use https protocol
letsencrypt['contact_emails'] = ['xxxx@magnas.com' 'xxx@magnas.com'] # This should be an array of email addresses to add as contacts
/var/opt/gitlab/nginx/conf/gitlab-http.conf
## Redirects all HTTP traffic to the HTTPS host
server {
listen 0.0.0.0:80;
listen [::]:80;
server_name gitlab.magnas.com;
server_tokens off; ## Don't show the nginx version number, a security best practice
location / {
return 301 https://gitlab.magnas.com:443$request_uri;
}
# health checks configuration
include /var/opt/gitlab/nginx/conf/gitlab-health.conf;
access_log /var/log/gitlab/nginx/gitlab_access.log gitlab_access;
error_log /var/log/gitlab/nginx/gitlab_error.log error;
}
server {
listen 0.0.0.0:443 ssl http2;
listen [::]:443 ssl http2;
server_name gitlab.magnas.com;
server_tokens off; ## Don't show the nginx version number, a security best practice
## Increase this if you want to upload large attachments
## Or if you want to accept large git objects over http
client_max_body_size 250m;
## Strong SSL Security
## https://raymii.org/s/tutorials/Strong_SSL_Security_On_nginx.html & https://cipherli.st/
ssl_certificate /etc/gitlab/ssl/gitlab.magnas.com.crt;
ssl_certificate_key /etc/gitlab/ssl/gitlab.magnas.com.key;
ssl_client_certificate /etc/gitlab/ssl/gitlab.magnas.com.crt;
# GitLab needs backwards compatible ciphers to retain compatibility with Java IDEs
ssl_ciphers 'ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:AES256-GCM-SHA384';
ssl_protocols TLSv1.2 TLSv1.3;
ssl_prefer_server_ciphers off;
ssl_session_cache shared:SSL:10m;
ssl_session_tickets off;
ssl_session_timeout 1d;
## Real IP Module Config
## http://nginx.org/en/docs/http/ngx_http_realip_module.html
## HSTS Config
## https://www.nginx.com/blog/http-strict-transport-security-hsts-and-nginx/
add_header Strict-Transport-Security "max-age=63072000";
# Rails sets a default policy of strict-origin-when-cross-origin, so
# hide that and just send the one we've configured for nginx
proxy_hide_header Referrer-Policy;
add_header Referrer-Policy strict-origin-when-cross-origin;
## Individual nginx logs for this GitLab vhost
access_log /var/log/gitlab/nginx/gitlab_access.log gitlab_access;
error_log /var/log/gitlab/nginx/gitlab_error.log error;
if ($http_host = "") {
set $http_host_with_default "gitlab.magnas.com";
}
if ($http_host != "") {
set $http_host_with_default $http_host;
}
gzip on;
gzip_static on;
gzip_comp_level 2;
gzip_http_version 1.1;
gzip_vary on;
gzip_disable "msie6";
gzip_min_length 250;
gzip_proxied no-cache no-store private expired auth;
gzip_types text/plain text/css application/x-javascript text/xml application/xml application/xml+rss text/javascript application/json;
## https://github.com/gitlabhq/gitlabhq/issues/694
## Some requests take more than 30 seconds.
proxy_read_timeout 3600;
proxy_connect_timeout 300;
proxy_redirect off;
proxy_http_version 1.1;
proxy_set_header Host $http_host_with_default;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
proxy_set_header X-Forwarded-Proto https;
proxy_set_header X-Forwarded-Ssl on;
location ~ (/api/v\d/jobs/\d+/artifacts$|\.git/git-receive-pack$|\.git/gitlab-lfs/objects|\.git/info/lfs/objects/batch$) {
proxy_cache off;
proxy_pass http://gitlab-workhorse;
proxy_request_buffering off;
}
location /-/grafana/ {
proxy_pass http://localhost:3000/;
}
location = /-/kubernetes-agent/ {
proxy_pass http://localhost:8150/;
}
location /-/kubernetes-agent/k8s-proxy/ {
proxy_pass http://localhost:8154/;
}
# health checks configuration
include /var/opt/gitlab/nginx/conf/gitlab-health.conf;
location / {
proxy_cache off;
proxy_pass http://gitlab-workhorse;
}
location /assets {
add_header X-Content-Type-Options nosniff;
proxy_cache gitlab;
proxy_pass http://gitlab-workhorse;
}
error_page 404 /404.html;
error_page 500 /500.html;
error_page 502 /502.html;
location ~ ^/(404|500|502)(-custom)?\.html$ {
root /opt/gitlab/embedded/service/gitlab-rails/public;
internal;
}
/var/log/gitlab/nginx/error.log
/gitlab/nginx/conf/gitlab-http.conf:34
2022/08/02 08:58:41 [emerg] 3484915#0: no "ssl_certificate" is defined for the "listen ... ssl" directive in /var/opt/gitlab/nginx/conf/gitlab-http.conf:34
2022/08/02 08:58:42 [emerg] 3484932#0: no "ssl_certificate" is defined for the "listen ... ssl" directive in /var/opt/gitlab/nginx/conf/gitlab-http.conf:34
2022/08/02 08:58:43 [emerg] 3485005#0: no "ssl_certificate" is defined for the "listen ... ssl" directive in /var/opt/gitlab/nginx/conf/gitlab-http.conf:34
2022/08/02 08:58:44 [warn] 3485019#0: conflicting server name "gitlab.magnas.com" on 0.0.0.0:80, ignored
2022/08/02 08:58:44 [warn] 3485019#0: conflicting server name "gitlab.magnas.com" on 0.0.0.0:443, ignored
2022/08/02 10:16:56 [warn] 3494210#0: conflicting server name "gitlab.magnas.com" on 0.0.0.0:443, ignored
2022/08/02 10:24:13 [warn] 3495595#0: conflicting server name "gitlab.magnas.com" on 0.0.0.0:80, ignored
2022/08/02 10:24:13 [warn] 3495595#0: conflicting server name "gitlab.magnas.com" on 0.0.0.0:443, ignored
2022/08/02 11:31:36 [warn] 3504813#0: conflicting server name "gitlab.magnas.com" on 0.0.0.0:80, ignored
2022/08/02 11:31:36 [warn] 3504813#0: conflicting server name "gitlab.magnas.com" on 0.0.0.0:443, ignored
2022/08/02 11:36:57 [warn] 3506143#0: conflicting server name "gitlab.magnas.com" on 0.0.0.0:80, ignored
2022/08/02 11:36:57 [warn] 3506143#0: conflicting server name "gitlab.magnas.com" on 0.0.0.0:443, ignored
2022/08/02 11:37:02 [warn] 3506221#0: conflicting server name "gitlab.magnas.com" on 0.0.0.0:80, ignored
2022/08/02 11:37:02 [warn] 3506221#0: conflicting server name "gitlab.magnas.com" on 0.0.0.0:443, ignored
on Server hosting Gitlab-runner (RHEL 9)
/etc/gitlab-runner/conf.toml
concurrent = 1
check_interval = 0
[session_server]
session_timeout = 1800
[[runners]]
name = "rhel-runner-2"
url = "https://gitlab.magnas.com"
token = "rxF8jJtMTJdBgayB-Vbe"
tls-cert-file = "/etc/gitlab-runner/certs/gitlab.magnas.com.crt"
tls-key-file = "/etc/gitlab-runner/certs/gitlab.magnas.com.key"
executor = "docker"
clone_url = "http://10.30.36.240"
[runners.custom_build_dir]
[runners.cache]
[runners.cache.s3]
[runners.cache.gcs]
[runners.cache.azure]
[runners.docker]
tls_verify = false
insecure = true
image = "ruby:2.7"
privileged = false
disable_entrypoint_overwrite = false
oom_kill_disable = false
disable_cache = false
volumes = ["/var/run/docker.sock:/var/run/docker.sock", "/cache"]
shm_size = 0
/etc/docker/daemon.json
I was getting HTTPS error “Unknown Certificate” so i found a workaround and added this file.
{"insecure-registries" : ["gitlab.magnas.com:80"]}
/etc/default/docker
DOCKER_OPTS="--config-file=/etc/docker/daemon.json"
What troubleshooting steps are yet to be taken? Any help is well much appreciated.
Thanks!