Hello GitLab-Community,
I run a self-hosted GitLab using docker-compose (docker-compose.yml posted below), which is installed on a computer in my living room. I am connected to the internet via DS-Lite. The letsencrypt support is turned on and GitLab is availble from the Web via https://treffer-technologies.feste-ip.net/.
About a week ago my gitlab-runner suddently failed to connect due to a certification error. I suspected a hickup with the auto-renewal of the letsencrypt certificate. After 10+ hours of testing out numerous hints from the GitLab and LetsEncrypt Community without success I started to investigate this error in gitlab itself, not the runner:
================================================================================
Error executing action `create` on resource 'letsencrypt_certificate[treffer-technologies.feste-ip.net]'
================================================================================
RuntimeError
------------
acme_certificate[staging] (letsencrypt::http_authorization line 43) had an error: RuntimeError: ruby_block[create certificate for treffer-technologies.feste-ip.net] (letsencrypt::http_authorization line 110) had an error: RuntimeError: [treffer-technologies.feste-ip.net] Validation failed, unable to request certificate, Errors: [{url: https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/3362084934/7XFM8w, status: invalid, error: {"type"=>"urn:ietf:params:acme:error:unauthorized", "detail"=>"The key authorization file from the server did not match this challenge \"RKPzvsY_ccw6aqr_U605vhOjrcv3yasjCXkX1xQ4bMI.7J-3s-TQuS3G4xwqmXCog5Psd_ed5r2wI7zTtpf8Tuo\" != \"RKPzvsY_ccw6aqr_U605vhOjrcv3yasjCXkX1xQ4bMI.oBNd5smO5vYJ4JXg-7VH8ZPOOgURgOqPb-Ffq1bGeKA\"", "status"=>403}} ]
As you can see, the filename of the challenge itself is identical, but the hash-value after the dot is not. What also struck me as odd was that my GitLab was still reachable with a good locking certificate from letsencrypt, but the auto-generated certificate in /etc/gitlab/config/ssl is self-signed (Issuer: CN = treffer-technologies.feste-ip.net).
As I mentioned above my internet provider uses DS-Lite. To workaround that and be able to have a fix URL for my gitlab I am using the DDNS and portmapper service of feste-ip.net (german for ‘fixed-ip’), hence the URL my GitLab is reachable from. As per docker-compose.yml, the container listens to port 30080 and 30443. But for letsencrypt support in gitlab to work, ports 80 and 443 have to be reachable. Therefore, I use HTTP(s)-Proxies from feste-ip.net, which allows me to map 80 → 30080 and 443 → 30443. With that http://treffer-technologies.feste-ip.net/ and https://treffer-technologies.feste-ip.net/ work.
Now I found out that feste-ip.net have setup their own letsencrypt for HTTPS-Proxy and they will not turn it off for me. This explains the non-matching hash-values since their proxy-server responde to the acme-challenge and holds the letsencrypt certificate.
This is how I envision my setup in my head:
After all that explanation, here my question:
How would your setup look like with these circumstances (gitlab publically available and hence using ssl, works with gitlab-runner, stuck on DS-Lite, very low budget)?
a) deactivate letsencrypt support in gitlab and rely on feste-ips letsencrypt
b) deactivate letsencrypt support in gitlab and use a self-signed certificate
c) something else
I’m pretty sure that gitlab-runner will give me a headache again once this is sorted out. For example, what certificate do I have to pass it in ordner to make a connection and later be able to use git-commands? But I guess I’ll cross that bridge when I reach it.
version: '3.7'
services:
postgresql:
...
redis:
...
plantuml:
...
gitlab:
image: 'gitlab/gitlab-ce:latest'
restart: always
hostname: 'treffer-technologies.feste-ip.net'
logging:
...
links:
- postgresql:postgresql
- redis:redis
environment:
GITLAB_OMNIBUS_CONFIG: |
...
# nginx - ssl
nginx['redirect_http_to_https'] = true
nginx['redirect_http_to_https_port'] = 80
nginx['client_max_body_size'] = "350m"
# other
gitlab_rails['gitlab_shell_ssh_port'] = 22
# https://docs.gitlab.com/omnibus/settings/ssl.html#lets-encrypt-integration
letsencrypt['enable'] = true
letsencrypt['auto_renew'] = true
letsencrypt['auto_renew_hour'] = "12"
letsencrypt['auto_renew_minute'] = "30"
letsencrypt['auto_renew_day_of_month'] = "*/15"
external_url 'https://treffer-technologies.feste-ip.net'
...
ports:
- "30080:80"
- "30022:22"
- "30443:443"
volumes:
- /opt/gitlab/config:/etc/gitlab:rw
- /opt/gitlab/log:/var/log/gitlab:rw
- /opt/gitlab/data:/var/opt/gitlab:rw
depends_on:
- postgresql
- redis
runner:
image: 'gitlab/gitlab-runner:ubuntu'
restart: always
logging:
driver: "json-file"
options:
max-size: "10m"
max-file: "10"
volumes:
- /opt/gitlab-runner/config:/etc/gitlab-runner
- /var/run/docker.sock:/var/run/docker.sock
depends_on:
- gitlab
