GitLab restart failed. PostgreSQL and Sidekiq stuck. 502 errors

GitLab has been installed and running fine for years, currently on v12.10. Nothing has changed recently. Currently it is down.

About 10 days ago, some users reported 502 errors (only affected some users). Eventually I ended up rebooting the server, and everything ran fine again for a few days. I noticed another 502 today, and when I tried to gitlab-ctl restart, postgresql and sidekiq wouldn’t go down:

timeout: run: postgresql: (pid 2442) 309841s, want down
...
timeout: run: sidekiq: (pid 2459) 309873s, want down, got TERM

If I try to start the dbconsole, I get: FATAL: the database system is shutting down (PG::ConnectionBad)

Any advice on what I should do next?

Thanks in advance for any help.
Patrick

Edit: The machine was running with 2 processors. It’s been bumped up to 4 and rebooted.
It’s running good now, but let’s give it some time and see if the 502s come back…

First I would check what your disk space looks like, are any of your partitions short? Also, check when it happens, check your RAM levels. How much ram do you have? Has it started using swap?

Whilst minimum requirements say 4cpu and 4gb of ram, I find it works best with 8gb of ram, but this might need to be increased especially when there are more and more projects/users. So I would check that you meet the minimum requirements, but also depending on if your server started using swap or not, you might want to think about increasing your ram.