I have never been able to seamlessly upgrade Gitlab using the system’s package manager, which in my case is apt
. Server runs Ubuntu 18.04 LTS and after upgrading packages, Gitlab’s update always hangs.
Whenever this happened in the past, I would simply:
- follow the steps in the documentation on backing up gitlab’s data
- shutdown gitlab with
gitlab-ctl stop
andsystemctl stop gitlab-runsvdir
- remove the package using the OS’s package manager:
apt remove gitlab-ee
- remove
/opt/gitlab
- remove
/var/opt/gitlab
(after creating a manual backup copy) - reinstall gitlab using the package manager
- follow the steps in the documentation on restoring data
- I think I even
gitlab-ctl stop
thengitlab-ctl reconfigure
before finallygitlab-ctl start
Taking the steps above usually led to two bundle
processes running for a few seconds during which I’d get 502s whenever trying to access Gitlab in the browser, however eventually bundle
would conclude and I’d be able to regain access to my local Gitlab instance.
This specific issue happened while upgrading a local instance of Gitlab (v11.6.0) to v11.9.1. After that initial time when bundle
runs and the 502s that normally ensue, this time I’m getting persistent 500s. This is to say that I cannot access Gitlab at all even after installing a pristine copy of Gitlab. I can only access Gitlab if I don’t restore my data – so, if I skip step 7 above – but as soon as the restore is done, 500s abound.
Somewhat strangely, nginx errors logs are empty and its access logs contain unhelpful entries similar to the following:
127.0.0.1 - - [31/Mar/2019:10:40:23 +0100] "GET /help HTTP/1.1" 500 2926 "" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Safari/537.36"
127.0.0.1 - - [31/Mar/2019:10:40:23 +0100] "GET /favicon.ico HTTP/1.1" 500 2926 "http://localhost:9000/help" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Safari/537.36"
127.0.0.1 - - [31/Mar/2019:10:44:03 +0100] "GET /help HTTP/1.1" 500 2926 "" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Safari/537.36"
127.0.0.1 - - [31/Mar/2019:10:44:04 +0100] "GET /favicon.ico HTTP/1.1" 500 2926 "http://localhost:9000/help" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Safari/537.36"
Can someone please help me debug this?
EDIT: can also confirm that downgrading to 11.6.0 fixes the issue and I’m able to regain access. This suggests there’s something wrong with the 11.9.1 version.