Gitlab not working after reboot

Hi guys, i have a Gitlab hosted on Ubuntu.

I got a user saying that the server must be rebooted because of 500 internal server error.

Before rebooting the http://192.168.5.55 was working

After reboot the page is not showing up.

gitlab-ctl status seems to be fine: AFAIK

ok: run: gitaly: (pid 5392) 1s
ok: run: gitlab-monitor: (pid 5416) 0s
ok: run: gitlab-workhorse: (pid 5421) 1s
ok: run: logrotate: (pid 5431) 0s
ok: run: nginx: (pid 5512) 0s
ok: run: node-exporter: (pid 5520) 1s
ok: run: postgres-exporter: (pid 5530) 0s
ok: run: postgresql: (pid 5539) 1s
ok: run: prometheus: (pid 5544) 0s
ok: run: redis: (pid 5559) 0s
ok: run: redis-exporter: (pid 5565) 0s
ok: run: sidekiq: (pid 5572) 1s
ok: run: unicorn: (pid 5582) 0s

ss -ntlp nginx

Is showing that the nginx is listening and is binding to port 80

I have rebooted the server 3x already but still the site is not coming up.

I check the configuration everything seems to be the same. The server is behind VPN.

Any ideas is greatly welcome. Thank you.

Generally after a reboot/restart of gitlab you need to give it a minute or two for the system to start everything up, even if gitlab-ctl status shows all services running, because processes like ruby/rails etc have to get going which takes a bit of time.

If you are still having issues with errors after a few minutes, then try:

gitlab-ctl reconfigure

and then:

gitlab-ctl restart

and see if that helps. Maybe something needs fixing in the config, usually after upgrades, etc, this is what you do to get it going if it fails to work, but I’ve never had such situation yet.

If still failing, check the logs, because we’d need to see if any errors are showing somewhere to be able to debug further.

1 Like

i did check the log, and it’s very funny. error log is zero bytes. which i believe is correct because the server was running fine, i just rebooted it to check if the 500 internal server error that the user was complaining will resolve the issue.

nothing harm to run gitlab-ctl reconfigure even though no changes has been done on the config?

Thanks.

No, no harm in running it, even if config hasn’t changed. It sometimes can rectify issues if something isn’t working properly after, eg: upgrade, or something else.

1 Like

Hi Ian, thanks for the update.

gitlab-ctl reconfigure didn’t work

even though it says reconfigured successfully

but i notice every time i reboot the system, the gitlab-http.conf has this value:

server {

to listen ip.add.ress:80:80

and nginx will not run

if i change it to:
listen ip.add.ress:80

then restart nginx still the gitlab page still not showing

any ideas why that line keep changing?

thanks

Port 80 is open in ufw status

gitlab-ctl status shows all are running but somehow GitLab is unreachable in my browser

Seems strange that it’s replacing in the config like that. Can you post everything from your gitlab.rb relating to nginx that is uncommented. It will have “nginx” on the config files in particular. For example:

cat /etc/gitlab/gitlab.rb | grep -i nginx | grep -i listen
nginx['listen_addresses'] = ['x.x.x.x', '[::]']

of course Ive masked my ip above with x.x.x.x. But yours should have a valid IP in here. If there are any other nginx lines uncommented, do review the file and paste here as it might explain why it’s changing. Also show the external_url please from config file. Maybe :80 is doubled up somewhere.

Also check /var/log/gitlab/nginx/gitlab_error.log and see if anything useful in here. Also for comparison, post results from:

netstat -tunlp

this will show us what ports are listening and could help when I compare against mine if something is missing.

1 Like

Hi Ian,

Thanks for the update. Please find below output.

ubuntu# cat /etc/gitlab/gitlab.rb | grep -i nginx | grep -i listen

# nginx['listen_addresses'] = ['*', '[::]']
##! Docs: https://docs.gitlab.com/omnibus/settings/nginx.html#setting-the-nginx-listen-port
# nginx['listen_port'] = nil
# nginx['listen_https'] = nil
nginx['listen_addresses'] = ['192.168.xx.xx:80']

ubuntu# netstat -tunlp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 127.0.0.1:9236          0.0.0.0:*               LISTEN      22596/gitaly
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1469/sshd
tcp        0      0 0.0.0.0:25              0.0.0.0:*               LISTEN      1615/master
tcp        0      0 127.0.0.1:8090          0.0.0.0:*               LISTEN      22626/gitlab-pages
tcp        0      0 0.0.0.0:8060            0.0.0.0:*               LISTEN      1419/nginx
tcp        0      0 127.0.0.1:9121          0.0.0.0:*               LISTEN      22826/redis_exporte
tcp        0      0 127.0.0.1:8065          0.0.0.0:*               LISTEN      22744/mattermost
tcp        0      0 127.0.0.1:9090          0.0.0.0:*               LISTEN      22807/prometheus
tcp        0      0 127.0.0.1:9187          0.0.0.0:*               LISTEN      22783/postgres_expo
tcp        0      0 127.0.0.1:5000          0.0.0.0:*               LISTEN      22832/registry
tcp        0      0 127.0.0.1:9100          0.0.0.0:*               LISTEN      22766/node_exporter
tcp        0      0 127.0.0.1:9229          0.0.0.0:*               LISTEN      22647/gitlab-workho
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      1419/nginx
tcp        0      0 127.0.0.1:8080          0.0.0.0:*               LISTEN      10991/config.ru
tcp        0      0 127.0.0.1:9168          0.0.0.0:*               LISTEN      22622/ruby
tcp        0      0 127.0.0.1:8082          0.0.0.0:*               LISTEN      22845/sidekiq 5.0.5
tcp6       0      0 :::22                   :::*                    LISTEN      1469/sshd
tcp6       0      0 :::25                   :::*                    LISTEN      1615/master
udp        0      0 0.0.0.0:68              0.0.0.0:*                           911/dhclient

You don’t need the :80 in the IP address field. Please remove this. It should just have the IP address, nothing more.

It should look like this:

nginx['listen_addresses'] = ['192.168.1.10', '[::]']

for example. Then run gitlab-ctl reconfigure and gitlab-ctl restart and see if it works.

1 Like

Hi iwalker, thanks for your help. What I did is to install a VM with Gitlab community version and compare the settings of gitlab.rb.
For whatever reason, the settings on giltab.rb was different from the newly installed Giltab on VM.
So, I tried to make it both similar and the problematic machine works with the same settings on the newly installed VM.
I am not sure why the previous settings was able to run, when it shouldn’t be running.
But then issue comes out after reboot.

Thank you for your help. Cheers!