hey im running a server in aws that is in a vpc. so the server has a private ip: 10.235.77.28
admin@ip-10-235-77-28:~/_volumes/gitlab/config$ sudo firewall-cmd --list-all
etldev_fd (active)
target: DROP
icmp-block-inversion: no
interfaces:
sources: 10.0.0.0/8
services:
ports: 22/tcp 80/tcp 443/tcp 2222/tcp
protocols:
forward: no
masquerade: yes
forward-ports:
port=80:proto=tcp:toport=8081:toaddr=
port=443:proto=tcp:toport=4431:toaddr=
source-ports:
icmp-blocks:
rich rules:
i have two containers running in the server (not through compose):
- nginx
- gitlab (omnibus)
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
e8de34ae9f7b docker.io/gitlab/gitlab-ee:15.8.5-ee.0 /assets/wrapper 19 hours ago Up 13 hours ago 0.0.0.0:2222->22/tcp, 0.0.0.0:8082->80/tcp, 0.0.0.0:4430->443/tcp, 0.0.0.0:5005->5005/tcp gitlab
6d70e90ef7b2 docker.io/library/nginx:1.23.2 nginx -g daemon o... 19 hours ago Up 12 hours ago 0.0.0.0:4431->4430/tcp, 0.0.0.0:8081->8080/tcp nginx.main
i set up on duckdns a domain that points to my private ip: wketl2.duckdns.org
i created a certificate for the server:
root@e8de34ae9f7b:/etc/gitlab/ssl# ls -la
total 16
drwxr-xr-x 2 root root 4096 May 4 16:59 .
drwxrwxr-x 4 root root 4096 May 4 17:22 ..
-rw------- 1 root root 1704 May 4 16:42 10.235.77.28+2-key.pem
-rw-r--r-- 1 root root 1578 May 4 16:42 10.235.77.28+2.pem
lrwxrwxrwx 1 root root 18 May 4 16:43 10.235.77.28.crt -> 10.235.77.28+2.pem
lrwxrwxrwx 1 root root 22 May 4 16:43 10.235.77.28.key -> 10.235.77.28+2-key.pem
lrwxrwxrwx 1 root root 18 May 4 16:57 registry.wketl2.duckdns.org.crt -> 10.235.77.28+2.pem
lrwxrwxrwx 1 root root 22 May 4 16:57 registry.wketl2.duckdns.org.key -> 10.235.77.28+2-key.pem
lrwxrwxrwx 1 root root 18 May 4 16:59 wketl2.duckdns.org.crt -> 10.235.77.28+2.pem
lrwxrwxrwx 1 root root 22 May 4 16:59 wketl2.duckdns.org.key -> 10.235.77.28+2-key.pem
then i added it as rootCA in the gitlab container:
root@e8de34ae9f7b:/# ln -s /etc/gitlab/ssl/10.235.77.28+2.pem /usr/local/share/ca-certificates/rootCA.pem
root@e8de34ae9f7b:/# ls -la /usr/local/share/ca-certificates/
total 8
drwxr-xr-x 2 root root 4096 May 4 17:33 .
drwxr-xr-x 4 root root 4096 Mar 30 18:26 ..
lrwxrwxrwx 1 root root 34 May 4 17:33 rootCA.pem -> /etc/gitlab/ssl/10.235.77.28+2.pem
in gitlab.rb:
external_url 'https://wketl2.duckdns.org/gitlab';
registry_external_url 'https://registry.wketl2.duckdns.org:5005'
### Settings used by GitLab application
gitlab_rails['registry_enabled'] = true
gitlab_rails['registry_host'] = "wketl2.duckdns.org"
gitlab_rails['registry_port'] = "5005"
gitlab_rails['registry_path'] = "/var/opt/gitlab/gitlab-rails/shared/registry"
nginx['ssl_certificate'] = "/etc/gitlab/ssl/wketl2.duckdns.org.crt"
nginx['ssl_certificate_key'] = "/etc/gitlab/ssl/wketl2.duckdns.org.key"
nginx['ssl_client_certificate'] = "/usr/local/share/ca-certificates/rootCA.pem"
in podman config:
[[registry]]
insecure=false
location="registry.wketl2.duckdns.org:5005"
make sure the status is good:
admin@ip-10-235-77-28:~/_volumes/gitlab/config$ podman exec -it gitlab /bin/bash -c 'gitlab-ctl status'
run: alertmanager: (pid 327) 47235s; run: log: (pid 318) 47235s
run: gitaly: (pid 20047) 3200s; run: log: (pid 316) 47235s
run: gitlab-exporter: (pid 321) 47235s; run: log: (pid 311) 47235s
run: gitlab-kas: (pid 22686) 1601s; run: log: (pid 309) 47235s
run: gitlab-workhorse: (pid 19886) 3248s; run: log: (pid 330) 47235s
run: logrotate: (pid 23100) 435s; run: log: (pid 326) 47235s
run: nginx: (pid 23887) 225s; run: log: (pid 325) 47235s
run: postgres-exporter: (pid 329) 47235s; run: log: (pid 320) 47235s
run: postgresql: (pid 328) 47235s; run: log: (pid 319) 47235s
run: prometheus: (pid 339) 47235s; run: log: (pid 331) 47235s
run: puma: (pid 22646) 1637s; run: log: (pid 312) 47235s
run: redis: (pid 342) 47235s; run: log: (pid 332) 47235s
run: redis-exporter: (pid 317) 47235s; run: log: (pid 310) 47235s
run: registry: (pid 22702) 1601s; run: log: (pid 308) 47235s
run: sidekiq: (pid 22618) 1645s; run: log: (pid 313) 47235s
run: sshd: (pid 28) 47246s; run: log: (pid 27) 47246s
when i visit the site, it works. also:
admin@ip-10-235-77-28:~/_volumes/gitlab/config$ curl -k https://10.235.77.28:4430/gitlab/
<html><body>You are being <a href="https://10.235.77.28:4430/gitlab/users/sign_in">redirected</a>.</body></html>admin@ip-10-235-77-28:~/_volumes/gitlab/config$ ^C
admin@ip-10-235-77-28:~/_volumes/gitlab/config$ curl --cacert ~/_volumes/gitlab/config/ssl/wketl2.duckdns.org.crt https://wketl2.duckdns.org:4430/gitlab
<html><body>You are being <a href="https://wketl2.duckdns.org:4430/gitlab/users/sign_in">redirected</a>.</body></html>admin@ip-10-235-77-28:~/_volumes/gitlab/config$ ^C
nginx on the host:
map $http_upgrade $connection_upgrade {
default upgrade;
"" close;
}
server {
listen 2222;
server_name 10.235.77.28;
location / {
proxy_pass http://10.235.77.28/gitlab/:2222;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
}
}
server {
listen 8080;
listen 4430 ssl http2;
listen [::]:4430 ssl http2;
server_name 10.235.77.28 wketl2.duckdns.org;
proxy_set_header X-Forwarded-For $proxy_protocol_addr; # To forward the original client's IP address
proxy_set_header X-Forwarded-Proto $scheme; # to forward the original protocol (HTTP or HTTPS)
proxy_set_header Host $host; # to forward the original host requested by the client
# SSL configuration
ssl_certificate /usr/apps/gitlab/config/ssl/10.235.77.28+2.pem;
ssl_certificate_key /usr/apps/gitlab/config/ssl/10.235.77.28+2-key.pem;
location /gitlab{
access_log /var/log/nginx/gitlab_access.log;
error_log /var/log/nginx/gitlab_error.log;
client_max_body_size 0;
gzip off;
proxy_read_timeout 300;
proxy_connect_timeout 300;
proxy_pass https://10.235.77.28:4430;
proxy_set_header X-Script-Name /gitlab;
proxy_set_header Host $http_host;
proxy_redirect off;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
}
location / {
root /usr/share/nginx/html;
index index.html index.htm;
}
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root /usr/share/nginx/html;
}
}
try to log into the registry:
admin@ip-10-235-77-28:~/_volumes/gitlab/config$ podman login registry.wketl2.duckdns.org
Username: root
Password:
Error: error authenticating creds for "registry.wketl2.duckdns.org": error pinging docker registry registry.wketl2.duckdns.org: Get "https://registry.wketl2.duckdns.org/v2/": dial tcp 10.235.77.28:443: connect: connection refused
check the registry connection with a token i made:
admin@ip-10-235-77-28:~/_volumes/gitlab/config$ curl -v -H "Authorization: Bearer glpat-1BP-cs7CRwyLMZi_JY61" https://registry.wketl2.duckdns.org:5005/v2/?
* Trying 10.235.77.28:5005...
* Connected to registry.wketl2.duckdns.org (10.235.77.28) port 5005 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
* CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
* subject: O=mkcert development certificate; OU=admin@ip-10-235-77-28 (Debian)
* start date: May 4 16:42:24 2023 GMT
* expire date: Aug 4 16:42:24 2025 GMT
* subjectAltName: host "registry.wketl2.duckdns.org" matched cert's "registry.wketl2.duckdns.org"
* issuer: O=mkcert development CA; OU=admin@ip-10-235-77-28 (Debian); CN=mkcert admin@ip-10-235-77-28 (Debian)
* SSL certificate verify ok.
> GET /v2/? HTTP/1.1
> Host: registry.wketl2.duckdns.org:5005
> User-Agent: curl/7.74.0
> Accept: */*
> Authorization: Bearer glpat-1BP-cs7CRwyLMZi_JY61
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Mark bundle as not supporting multiuse
< HTTP/1.1 401 Unauthorized
< Server: nginx
< Date: Thu, 04 May 2023 18:03:48 GMT
< Content-Type: application/json
< Content-Length: 87
< Connection: keep-alive
< Docker-Distribution-Api-Version: registry/2.0
< Www-Authenticate: Bearer realm="https://wketl2.duckdns.org/gitlab/jwt/auth",service="container_registry",error="invalid_token"
< X-Content-Type-Options: nosniff
<
{"errors":[{"code":"UNAUTHORIZED","message":"authentication required","detail":null}]}
* Connection #0 to host registry.wketl2.duckdns.org left intact
compare without port:
admin@ip-10-235-77-28:~$ curl -v https://registry.wketl2.duckdns.org/v2/?
* Trying 10.235.77.28:443...
* connect to 10.235.77.28 port 443 failed: Connection refused
* Failed to connect to registry.wketl2.duckdns.org port 443: Connection refused
* Closing connection 0
curl: (7) Failed to connect to registry.wketl2.duckdns.org port 443: Connection refused
checking open ssl:
admin@ip-10-235-77-28:~/_volumes/gitlab/config$ openssl s_client -showcerts -connect registry.wketl2.duckdns.org:5005
CONNECTED(00000003)
depth=1 O = mkcert development CA, OU = admin@ip-10-235-77-28 (Debian), CN = mkcert admin@ip-10-235-77-28 (Debian)
verify return:1
depth=0 O = mkcert development certificate, OU = admin@ip-10-235-77-28 (Debian)
verify return:1
---
Certificate chain
0 s:O = mkcert development certificate, OU = admin@ip-10-235-77-28 (Debian)
i:O = mkcert development CA, OU = admin@ip-10-235-77-28 (Debian), CN = mkcert admin@ip-10-235-77-28 (Debian)
-----BEGIN CERTIFICATE-----
blahblah
-----END CERTIFICATE-----
---
Server certificate
subject=O = mkcert development certificate, OU = admin@ip-10-235-77-28 (Debian)
issuer=O = mkcert development CA, OU = admin@ip-10-235-77-28 (Debian), CN = mkcert admin@ip-10-235-77-28 (Debian)
---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 1685 bytes and written 399 bytes
Verification: OK
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
Protocol : TLSv1.3
Cipher : TLS_AES_256_GCM_SHA384
Session-ID: EF18BE0F017DC7A0202EB232F9A679543F0475685AA54F7A9EC8BB7B931BA8C5
Session-ID-ctx:
Resumption PSK: AFEDE128C827878BB022F3E3F2B92D6D0EAE7774F44652E2B497EAA33DEB96044D9D4F9BF54E6518E956829FD75FD6F8
PSK identity: None
PSK identity hint: None
SRP username: None
TLS session ticket lifetime hint: 86400 (seconds)
TLS session ticket:
0000 - f6 2d bd f2 1b 60 a6 19-b8 52 88 b2 3c 04 36 bf .-...`...R..<.6.
0010 - 9b dd 0d f6 13 22 5e a4-de 0f 25 95 5b 40 d1 37 ....."^...%.[@.7
Start Time: 1683222028
Timeout : 7200 (sec)
Verify return code: 0 (ok)
Extended master secret: no
Max Early Data: 0
---
read R BLOCK
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
Protocol : TLSv1.3
Cipher : TLS_AES_256_GCM_SHA384
Session-ID: 501DE98CF0EEBBAFF515482226F57EE8F7A85BCD81E95A97823B75424D72C3E1
Session-ID-ctx:
Resumption PSK: 0475304292E4AA50D42F309F915DCF4024260DE138F65A231A7FD8476561BC47383361FEA2CE080845AC3C03D5B35BA3
PSK identity: None
PSK identity hint: None
SRP username: None
TLS session ticket lifetime hint: 86400 (seconds)
TLS session ticket:
0000 - 51 91 01 18 06 6a 38 b9-e6 77 c7 fe 5b 81 28 5d Q....j8..w..[.(]
0010 - d4 fa 35 a4 e8 15 3f 81-67 6e 4d 4a a7 0a 92 a6 ..5...?.gnMJ....
Start Time: 1683222028
Timeout : 7200 (sec)
Verify return code: 0 (ok)
Extended master secret: no
Max Early Data: 0
---
read R BLOCK
check the logs:
admin@ip-10-235-77-28:~/_volumes/gitlab/config$ podman exec -it gitlab /bin/bash -c 'gitlab-ctl tail registry'
2023-05-04_17:15:37.85411 time="2023-05-04T17:15:37.853Z" level=info msg="Starting upload purge in 13m0s" environment=production go_version=go1.18.7 instance_id=5ec95b7a-5f40-42b4-a756-abb1bf8c057b service=registry version=v3.63.0-gitlab
2023-05-04_17:17:49.44713 {"content_type":"application/json","correlation_id":"01GZKWA905H4F2K45M9PNX0AER","duration_ms":1,"host":"registry.wketl2.duckdns.org:5005","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"127.0.0.1:37564","remote_ip":"127.0.0.1","status":401,"system":"http","time":"2023-05-04T17:17:49.447Z","ttfb_ms":1,"uri":"/v2/","user_agent":"containers/5.10.3 (github.com/containers/image)","written_bytes":87}
2023-05-04_17:28:37.85501 time="2023-05-04T17:28:37.854Z" level=info msg="PurgeUploads starting: olderThan=2023-04-27 17:28:37.854872823 +0000 UTC m=-604019.970952336, actuallyDelete=true"
2023-05-04_17:28:37.85508 time="2023-05-04T17:28:37.855Z" level=info msg="Purge uploads finished. Num deleted=0, num errors=1"
2023-05-04_17:28:37.85509 time="2023-05-04T17:28:37.855Z" level=info msg="Starting upload purge in 24h0m0s" environment=production go_version=go1.18.7 instance_id=5ec95b7a-5f40-42b4-a756-abb1bf8c057b service=registry version=v3.63.0-gitlab
2023-05-04_17:42:02.60003 {"content_type":"application/json","correlation_id":"01GZKXPM36F639KEZ0XJ0H4CMT","duration_ms":1,"host":"registry.wketl2.duckdns.org:5005","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"127.0.0.1:51358","remote_ip":"127.0.0.1","status":401,"system":"http","time":"2023-05-04T17:42:02.599Z","ttfb_ms":1,"uri":"/v2/?","user_agent":"curl/7.74.0","written_bytes":87}
2023-05-04_17:43:13.69489 {"content_type":"application/json","correlation_id":"01GZKXRSGXG8P8CXYQ02WJG3GP","duration_ms":1,"host":"registry.wketl2.duckdns.org:5005","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"127.0.0.1:36166","remote_ip":"127.0.0.1","status":401,"system":"http","time":"2023-05-04T17:43:13.694Z","ttfb_ms":1,"uri":"/v2/?","user_agent":"curl/7.74.0","written_bytes":87}
2023-05-04_18:01:33.52179 {"content_type":"application/json","correlation_id":"01GZKYTBJG2256ZZ4WEQWCXB97","duration_ms":1,"host":"registry.wketl2.duckdns.org:5005","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"127.0.0.1:45486","remote_ip":"127.0.0.1","status":401,"system":"http","time":"2023-05-04T18:01:33.521Z","ttfb_ms":1,"uri":"/v2/?","user_agent":"curl/7.74.0","written_bytes":87}
2023-05-04_18:02:45.79437 {"content_type":"application/json","correlation_id":"01GZKYWJ51CE3AMZGQ0172T97H","duration_ms":1,"host":"registry.wketl2.duckdns.org:5005","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"127.0.0.1:35564","remote_ip":"127.0.0.1","status":401,"system":"http","time":"2023-05-04T18:02:45.794Z","ttfb_ms":1,"uri":"/v2/?","user_agent":"curl/7.74.0","written_bytes":87}
2023-05-04_18:03:48.50385 {"content_type":"application/json","correlation_id":"01GZKYYFCPE5CBRBSH035SQ1ZT","duration_ms":1,"host":"registry.wketl2.duckdns.org:5005","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"127.0.0.1:50942","remote_ip":"127.0.0.1","status":401,"system":"http","time":"2023-05-04T18:03:48.503Z","ttfb_ms":0,"uri":"/v2/?","user_agent":"curl/7.74.0","written_bytes":87}
really out of my depth here…