GitLab / Letsencrypt didn't renew certificate. Nginx won't accept HTTPS anymore

How to Use GitLab
1

Hey all, I’m hoping I’ve selected the correct area for this kind of query.

My server’s SSL expired despite being setup for letsencrypt.

I’ve gone through a lot of troubleshooting without success and I must be missing something. Hopefully not too obvious.

sudo gitlab-ctl reconfigure

sudo gitlab-ctl renew-le-certs

Both of these didn’t fix the issue.
Here’s my configuration:

gitlab.rb


## GitLab URL

external_url 'https://git.DOMAINREDACTED.com'

## GitLab NGINX

##! Docs: https://docs.gitlab.com/omnibus/settings/nginx.html

################################################################################

nginx['enable'] = true

nginx['client_max_body_size'] = '500000m'

nginx['redirect_http_to_https'] = false

nginx['redirect_http_to_https_port'] = 80

# nginx['ssl_certificate'] = "/etc/gitlab/ssl/git.DOMAINREDACTED.com.crt"
# nginx['ssl_certificate_key'] = "/etc/gitlab/ssl/git.DOMAINREDACTED.com.key"

# nginx['ssl_certificate'] = "/letsencrypt/live/git.DOMAINREDACTED.com/fullchain.pem"
# nginx['ssl_certificate_key'] = "/etc/letsencrypt/live/git.DOMAINREDACTED.com/privkey.pem"


################################################################################

# Let's Encrypt integration

################################################################################

letsencrypt['enable'] = true

# letsencrypt['contact_emails'] = ["support@DOMAINREDACTED.com", "NAME@DOMAINREDACTED.com"] # This should be an array of email addresses to add as contacts

# letsencrypt['group'] = 'root'

# letsencrypt['key_size'] = 2048

# letsencrypt['owner'] = 'root'

# letsencrypt['wwwroot'] = '/var/opt/gitlab/nginx/www'

# See http://docs.gitlab.com/omnibus/settings/ssl.html#automatic-renewal for more on these sesttings

letsencrypt['auto_renew'] = true

letsencrypt['auto_renew_hour'] = "2"

letsencrypt['auto_renew_minute'] = "30" # Should be a number or cron expression, if specified.

letsencrypt['auto_renew_day_of_month'] = "*/1"

Listening ports

systemd-r   832   systemd-resolve   13u  IPv4   18687      0t0  TCP 127.0.0.53:53 (LISTEN)
sshd       1097              root    3u  IPv4   20644      0t0  TCP *:22 (LISTEN)
sshd       1097              root    4u  IPv6   20646      0t0  TCP *:22 (LISTEN)
xrdp-sesm  1148              root    7u  IPv6   21960      0t0  TCP [::1]:3350 (LISTEN)
xrdp       1237              xrdp   11u  IPv6   23747      0t0  TCP *:3389 (LISTEN)
nginx      5248        gitlab-www    7u  IPv4 2478885      0t0  TCP *:80 (LISTEN)
nginx      5248        gitlab-www    8u  IPv4 2478886      0t0  TCP *:8060 (LISTEN)
nginx      5249        gitlab-www    7u  IPv4 2478885      0t0  TCP *:80 (LISTEN)
nginx      5249        gitlab-www    8u  IPv4 2478886      0t0  TCP *:8060 (LISTEN)
ruby       5274               git   18u  IPv4 2486194      0t0  TCP 127.0.0.1:8080 (LISTEN)
ruby       6053               git   18u  IPv4 2486194      0t0  TCP 127.0.0.1:8080 (LISTEN)
ruby       9248               git   18u  IPv4 2486194      0t0  TCP 127.0.0.1:8080 (LISTEN)
ruby       9831               git   18u  IPv4 2486194      0t0  TCP 127.0.0.1:8080 (LISTEN)
ruby      12781               git   18u  IPv4 2486194      0t0  TCP 127.0.0.1:8080 (LISTEN)
alertmana 32131 gitlab-prometheus    3u  IPv6 2478690      0t0  TCP *:9094 (LISTEN)
alertmana 32131 gitlab-prometheus    6u  IPv4 2479753      0t0  TCP 127.0.0.1:9093 (LISTEN)
gitaly    32169               git    8u  IPv4 2478769      0t0  TCP 127.0.0.1:9236 (LISTEN)
gitaly    32169               git    9u  IPv4 2478769      0t0  TCP 127.0.0.1:9236 (LISTEN)
gitlab-ex 32180               git    5u  IPv4 2478803      0t0  TCP 127.0.0.1:9168 (LISTEN)
gitlab-ex 32180               git    6u  IPv6 2478804      0t0  TCP [::1]:9168 (LISTEN)
gitlab-wo 32193               git    4u  IPv4 2478796      0t0  TCP 127.0.0.1:9229 (LISTEN)
grafana-s 32208 gitlab-prometheus    6u  IPv4 2479828      0t0  TCP 127.0.0.1:3000 (LISTEN)
nginx     32235              root    7u  IPv4 2478885      0t0  TCP *:80 (LISTEN)
nginx     32235              root    8u  IPv4 2478886      0t0  TCP *:8060 (LISTEN)
node_expo 32241 gitlab-prometheus    3u  IPv4 2479852      0t0  TCP 127.0.0.1:9100 (LISTEN)
postgres_ 32249       gitlab-psql    5u  IPv4 2479865      0t0  TCP 127.0.0.1:9187 (LISTEN)
prometheu 32343 gitlab-prometheus    6u  IPv4 2478932      0t0  TCP 127.0.0.1:9090 (LISTEN)
redis_exp 32360      gitlab-redis    3u  IPv4 2478973      0t0  TCP 127.0.0.1:9121 (LISTEN)
bundle    32374               git   19u  IPv4 2481562      0t0  TCP 127.0.0.1:8082 (LISTEN)

gitlab-ctl status

run: alertmanager: (pid 32131) 1831s; run: log: (pid 1339) 73673s
run: crond: (pid 32141) 1831s; run: log: (pid 1337) 73673s
run: gitaly: (pid 32163) 1830s; run: log: (pid 1351) 73673s
run: gitlab-exporter: (pid 32180) 1829s; run: log: (pid 1356) 73673s
run: gitlab-workhorse: (pid 32193) 1829s; run: log: (pid 1353) 73673s
run: grafana: (pid 32208) 1828s; run: log: (pid 1352) 73673s
run: logrotate: (pid 32221) 1828s; run: log: (pid 1328) 73673s
run: nginx: (pid 32235) 1828s; run: log: (pid 1350) 73673s
run: node-exporter: (pid 32241) 1827s; run: log: (pid 1335) 73673s
run: postgres-exporter: (pid 32249) 1827s; run: log: (pid 1357) 73673s
run: postgresql: (pid 32334) 1826s; run: log: (pid 1329) 73673s
run: prometheus: (pid 32343) 1826s; run: log: (pid 1355) 73673s
run: redis: (pid 32356) 1825s; run: log: (pid 1338) 73673s
run: redis-exporter: (pid 32360) 1825s; run: log: (pid 1349) 73673s
run: sidekiq: (pid 32374) 1823s; run: log: (pid 1354) 73673s
run: unicorn: (pid 6447) 1090s; run: log: (pid 1336) 73673s

Unsure what else I should provide right now.

Any help would be appreciated.

2

Hi,

which version of GitLab is involved here? You can fetch that e.g. at https://gitlabserver/help.
Is there anything suspicious in the production.log for Let’s Encrypt?

Cheers,
Michael

1 Like
3

Hey, the version is GitLab Enterprise Edition 12.6.2-ee running on Ubuntu 18.04.

Most recent from the /var/log/letsencrypt/letsencrypt.log is:

2020-01-09 04:32:49,921:DEBUG:acme.client:Storing nonce: 0101vPOSogQ59I8AiacOGpcuBsPDICJLjUFOvimcmEBag2c
2020-01-09 04:32:49,922:DEBUG:certbot.storage:Archive directory /etc/letsencrypt/archive/git.DOMAIN.com and live directory /etc/letsencrypt/live/git.DOMAIN.com created.
2020-01-09 04:32:49,923:DEBUG:certbot.storage:Writing certificate to /etc/letsencrypt/live/git.DOMAIN.com/cert.pem.
2020-01-09 04:32:49,923:DEBUG:certbot.storage:Writing private key to /etc/letsencrypt/live/git.DOMAIN.com/privkey.pem.
2020-01-09 04:32:49,923:DEBUG:certbot.storage:Writing chain to /etc/letsencrypt/live/git.DOMAIN.com/chain.pem.
2020-01-09 04:32:49,923:DEBUG:certbot.storage:Writing full chain to /etc/letsencrypt/live/git.DOMAIN.com/fullchain.pem.
2020-01-09 04:32:49,923:DEBUG:certbot.storage:Writing README to /etc/letsencrypt/live/git.DOMAIN.com/README.
2020-01-09 04:32:49,934:DEBUG:certbot.plugins.selection:Requested authenticator nginx and installer nginx
2020-01-09 04:32:49,945:DEBUG:certbot.cli:Var authenticator=nginx (set by user).
2020-01-09 04:32:49,945:DEBUG:certbot.cli:Var installer=nginx (set by user).
2020-01-09 04:32:49,952:DEBUG:certbot.storage:Writing new config /etc/letsencrypt/renewal/git.DOMAIN.com.conf.
2020-01-09 04:32:49,955:DEBUG:certbot.reporter:Reporting to user: Congratulations! Your certificate and chain have been saved at:
/etc/letsencrypt/live/git.DOMAIN.com/fullchain.pem
Your key file has been saved at:
/etc/letsencrypt/live/git.DOMAIN.com/privkey.pem
Your cert will expire on 2020-04-08. To obtain a new or tweaked version of this certificate in the future, simply run certbot again. To non-interactively renew *all* of your certificates, run "certbot renew"
2020-01-09 04:32:49,955:DEBUG:certbot.reporter:Reporting to user: If you like Certbot, please consider supporting our work by:

Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
Donating to EFF:                    https://eff.org/donate-le


2020-01-09 12:13:03,140:DEBUG:certbot.main:certbot version: 0.31.0
2020-01-09 12:13:03,141:DEBUG:certbot.main:Arguments: ['-q']
2020-01-09 12:13:03,142:DEBUG:certbot.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#manual,PluginEntryPoint#nginx,PluginEntryPoint#null,PluginEntryPoint#standalone,PluginEntryPoint#webroot)
2020-01-09 12:13:03,165:DEBUG:certbot.log:Root logging level set at 30
2020-01-09 12:13:03,166:INFO:certbot.log:Saving debug log to /var/log/letsencrypt/letsencrypt.log
2020-01-09 12:13:03,182:DEBUG:certbot.renewal:no renewal failures
4

Hi,

looks good to me. I’d guess that the old certificates are loaded into application memory, did you restart GitLab after renewing the certificates already?

Cheers,
Michael

5

Yeah, I’ve restarted the entire server, even. Nginx isn’t listening on port 443?

6

Tried manually adding the port 443 server { configuration to gitlab-http.conf
Doing gitlab-ctl hup nginx had no result. So I ran gitlab-ctl reconfigure which had the following:

Recipe: gitlab::nginx
  * directory[/var/opt/gitlab/nginx] action create
    - change mode from '0777' to '0750'
  * directory[/var/opt/gitlab/nginx/conf] action create
    - change mode from '0777' to '0750'
  * directory[/var/log/gitlab/nginx] action create (up to date)
  * link[/var/opt/gitlab/nginx/logs] action create (up to date)
  * template[/var/opt/gitlab/nginx/conf/gitlab-http.conf] action create
    - update content in file /var/opt/gitlab/nginx/conf/gitlab-http.conf from f59858 to eb6ff6
    --- /var/opt/gitlab/nginx/conf/gitlab-http.conf     2020-01-13 00:42:09.826352299 +0000
    +++ /var/opt/gitlab/nginx/conf/.chef-gitlab-http20200113-28153-rt2vsx.conf  2020-01-13 00:43:56.882118740 +0000
    @@ -121,27 +121,4 @@


     }
    -server {
    -    listen 443 ssl;
    -    server_name git.domainredacted.com;
    -
    -    modsecurity on;
    -    modsecurity_rules_file /etc/nginx/modsec/main.conf;
    -
    -    ssl on;
    -    ssl_certificate /home/redacted/Desktop/ssl-backup/git.domainredacted.com.crt;
    -    ssl_certificate_key /home/redacted/Desktop/ssl-backup/git.domainredacted.com.key;
    -
    -    client_max_body_size 0;
    -
    -    access_log /var/log/nginx/access_https_git.doaminredacted.com.log combined;
    -    error_log /var/log/nginx/error_https_git.doaminredacted.com.log;
    -
    -    location / {
    -    proxy_set_header Host $http_host;
    -    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    -    proxy_pass http://gitlab:80;
    -    }
    -}
    -
    - change mode from '0777' to '0644'

So it pulled the manual settings out, but still didn’t configure HTTPS.

7

Hi,

been wondering about the redirect_http_to_https setting. Is there a specific reason not to force 443 over 80 with regard of security? That may be the culprit here - try setting this to true and see what happens.

Cheers,
Michael

8

I’ve set redirect_http_to_https to true and the issue persists.