503 Service Temporarily Unavailable

Hi, my gitlab server on google cloud platform was working since 2019, yesterday without warning, it stopped working the web-client (503 Service Temporarily Unavailable), I guess the Nginx, although if I run the commands:

root@gitlab:/home/alejandrobalegno# sudo gitlab-ctl status
run: alertmanager: (pid 8276) 59721s; run: log: (pid 4771) 60145s
run: gitaly: (pid 8293) 59720s; run: log: (pid 4749) 60145s
run: gitlab-exporter: (pid 8301) 59719s; run: log: (pid 4763) 60145s
run: gitlab-workhorse: (pid 8307) 59719s; run: log: (pid 4761) 60145s
run: grafana: (pid 8324) 59719s; run: log: (pid 4769) 60145s
run: logrotate: (pid 17521) 2109s; run: log: (pid 4762) 60145s
run: nginx: (pid 28329) 0s; run: log: (pid 4753) 60145s
run: node-exporter: (pid 8351) 59718s; run: log: (pid 4770) 60145s
run: postgres-exporter: (pid 8361) 59717s; run: log: (pid 4755) 60145s
run: postgresql: (pid 8378) 59717s; run: log: (pid 4766) 60145s
run: prometheus: (pid 8388) 59717s; run: log: (pid 4754) 60146s
run: redis: (pid 8409) 59717s; run: log: (pid 4750) 60146s
run: redis-exporter: (pid 8414) 59717s; run: log: (pid 4773) 60146s
run: sidekiq: (pid 8426) 59715s; run: log: (pid 4760) 60146s
run: unicorn: (pid 8444) 59714s; run: log: (pid 4756) 60146s

I also tried with:

root@gitlab:/home/alejandrobalegno# gitlab-rake gitlab:check
Checking GitLab subtasks ...
Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 12.2.0 ? ... OK (12.2.0)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Internal API available: OK
Redis available via internal API: OK
gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes
Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab App ...
Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Projects have namespace: ... 
all projects say yes
Redis version >= 4.0.0? ... yes
Ruby version >= 2.5.3 ? ... yes (2.6.5)
Git version >= 2.22.0 ? ... yes (2.26.2)
Git user has default SSH configuration? ... yes
Active users: ... 6
Is authorized keys file accessible? ... yes
Elasticsearch version 5.6 - 6.x? ... skipped (elasticsearch is disabled)
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished

As far as I can see, everything seems to be in order.

Any ideas ?

Hi, usually when services misbehave you can try this:

gitlab-ctl restart

if that fails to make any difference, try:

systemctl restart gitlab-runsvdir

if that fails, then what is usually recommended after an upgrade for example is:

gitlab-ctl reconfigure
gitlab-ctl restart

hope that helps.

Thank you very much for replying, I tried all those commands and the web client is still not working.

Have you rebooted the instance? Just in case their is a process that isn’t responding or something, and that for some reason didn’t get killed and restarted with the commands from previous post.

If a reboot has been performed, then we’d need to check through the log files under /var/opt/gitlab to see if we can any errors or something that is causing the issue.

Thanks for sharing, I’m also want to know about it. Thanks

Guys, everything was fixed, the problem:

this server had docker enabled many containers running on gitlab default ports.