Upgaded to CE 11.3 and Gitlab will not start, 502 taking too long

I have Gitlab installed on a Centos 7 server, it has 8GB of memory and 2x 2.4 GHz cores.

I recently upgraded to Gitlab 11.3 from a pretty old version. I believe I had to upgrade to 10.8 and then 11.3.

Gitlab now won’t start, returning 502 forever and capping out both cores at 100%. The main processes seem to be sidekiq and unicorn

This would seem related https://docs.gitlab.com/ee/administration/troubleshooting/debug.html#502-gateway-timeout-after-unicorn-spins-at-100-cpu

I tried to follow the instructions in the link you sent, but it didn’t seem to help. Here’s what I did:

I ran gitlab-ctl status to get the unicorn pid:

run: unicorn: (pid 1590) 44s; run: log: (pid 611) 160999s`

I tried to run gdb -p 1590 and it would not attach, (no such process). I tried then gdb -p 611 and then call (void) rb_backtrace() and it said No symbol table is loaded. Use the "file" command.

I tried then to use htop to determine the pid of the unicorn process that was hogging all the CPU, which also said no such process.

Trying the following heading, Troubleshooting without affecting other users, wasn’t possible. It says to get some token from Profile Settings -> Access Tokens, but no URLs load without producing the 502.

I tried also increasing unicorn['worker_timeout'] = 300, which does not seem to have any effect.


Despite all of the above, I have still tried perusing the logs in /var/log/gitlab/unicorn/unicorn_stderr.log and only seem to get /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab.rb:31: warning: already initialized constant Gitlab::VERSION. Googling this hasn’t brought much info

Still haven’t gotten this working.

Should I try backing up, reinstalling, then restoring Gitlab? :expressionless: