Omnibus upgrade from ce-13.2.5 to ce-13.3.0 - Borken no service bound to http(s)

I was running gitlab-ce-13.2.5-ce.0.el8 on centos 8 and upgraded due to recommendation in the web ui.
I followed guide: Upgrade GitLab by using the GitLab package | GitLab and upgradeed to 13.3.0-ce.1.el8
# sudo yum install gitlab-ce

The upgrade seemed showed no errors - but is not working. I cannot find any process listening to port 80 or 443 using “netstat -lnp”. I have run sudo gitlab-ctl restart and rebooted the machine.

Other than standard settings I have set up self-signed certificate for https and modified /etc/gitlab/gitlab.rb accordingly.
external_url 'https://< server ip >’
nginx[‘redirect_http_to_https’] = true
nginx[‘ssl_certificate’] = “/etc/pki/tls/certs/cert.pem”
nginx[‘ssl_certificate_key’] = “/etc/pki/tls/certs/key.pem”
nginx[‘ssl_dhparam’] = “/etc/gitlab/dhparams.pem”

letsencrypt[‘enable’] = false

On a sidenote: I had disabled protocol http in firewall-cmd, but that got enabled again after upgrade - with no warning.

Where should I start looking for remedy?

UPDATE
Found this in /var/log/gitlab/nginx
2020/08/26 14:55:48 [emerg] 122519#0: cannot load certificate key "/etc/pki/tls/certs/key.pem": PEM_read_bio_PrivateKey() failed (SSL: error:2807106B:UI routines:UI_process:processing error:while reading strings error:0906406D:PEM routines:PEM_def_callback:problems getting password error:0907B068:PEM routines:PEM_read_bio_PrivateKey:bad password read)

According to documentation (NGINX settings | GitLab):

If the certificate.key file is password protected, NGINX will not ask for the password when you reconfigure GitLab. In that case, Omnibus GitLab will fail silently with no error messages. To remove the password from the key, run:
openssl rsa -in certificate_before.key -out certificate_after.key

Now, reconfigure GitLab:
sudo gitlab-ctl reconfigure
When the reconfigure finishes, your GitLab instance should be reachable at https://gitlab.example.com.

This solved the issue!
Would be nice if this kind of error was shown after upgrade!

Kind regards /K

I agree with your final statement. I’ve also previously spent time finding information in the nginx log.

1 Like

Another viable option might be to disable the use of password protected certificates altogether? It’s confusing that you can perform a initial set up of gitlab with password protected .pem when the upgrade does not support it. Just a thought. /K

1 Like

I am glad to hear this was solved, @Kermit! and I have taken your note about that error to the Product Team here at GitLab. Thanks!