Failed Gitlab 11.6.0 upgrade to 11.9.1 on Ubuntu 18.04 LTS

I have never been able to seamlessly upgrade Gitlab using the system’s package manager, which in my case is apt. Server runs Ubuntu 18.04 LTS and after upgrading packages, Gitlab’s update always hangs.

Whenever this happened in the past, I would simply:

  1. follow the steps in the documentation on backing up gitlab’s data
  2. shutdown gitlab with gitlab-ctl stop and systemctl stop gitlab-runsvdir
  3. remove the package using the OS’s package manager: apt remove gitlab-ee
  4. remove /opt/gitlab
  5. remove /var/opt/gitlab (after creating a manual backup copy)
  6. reinstall gitlab using the package manager
  7. follow the steps in the documentation on restoring data
  8. I think I even gitlab-ctl stop then gitlab-ctl reconfigure before finally gitlab-ctl start

Taking the steps above usually led to two bundle processes running for a few seconds during which I’d get 502s whenever trying to access Gitlab in the browser, however eventually bundle would conclude and I’d be able to regain access to my local Gitlab instance.

This specific issue happened while upgrading a local instance of Gitlab (v11.6.0) to v11.9.1. After that initial time when bundle runs and the 502s that normally ensue, this time I’m getting persistent 500s. This is to say that I cannot access Gitlab at all even after installing a pristine copy of Gitlab. I can only access Gitlab if I don’t restore my data – so, if I skip step 7 above – but as soon as the restore is done, 500s abound.

Somewhat strangely, nginx errors logs are empty and its access logs contain unhelpful entries similar to the following:

127.0.0.1 - - [31/Mar/2019:10:40:23 +0100] "GET /help HTTP/1.1" 500 2926 "" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Safari/537.36"
127.0.0.1 - - [31/Mar/2019:10:40:23 +0100] "GET /favicon.ico HTTP/1.1" 500 2926 "http://localhost:9000/help" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Safari/537.36"
127.0.0.1 - - [31/Mar/2019:10:44:03 +0100] "GET /help HTTP/1.1" 500 2926 "" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Safari/537.36"
127.0.0.1 - - [31/Mar/2019:10:44:04 +0100] "GET /favicon.ico HTTP/1.1" 500 2926 "http://localhost:9000/help" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Safari/537.36"

Can someone please help me debug this?

EDIT: can also confirm that downgrading to 11.6.0 fixes the issue and I’m able to regain access. This suggests there’s something wrong with the 11.9.1 version.