GitLab not starting after upgrading from Ubuntu 14.04

I was trying to upgrade the OS on a rather old VM running GitLab.
The existing OS version was Ubuntu 14,04, which was then upgraded to 16.04 and finally to 18.04 which it is running now:

lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 18.04.6 LTS
Release: 18.04
Codename: bionic

uname -a
Linux NOAZUGITLAB01 4.15.0-163-generic #171-Ubuntu SMP Fri Nov 5 11:55:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

However, GitLab stopped working after the upgrade to 16.04

gitlab-ctl status shows the following:
fail: alertmanager: runsv not running
fail: gitaly: runsv not running
fail: gitlab-monitor: runsv not running
fail: gitlab-workhorse: runsv not running
fail: logrotate: runsv not running
fail: nginx: runsv not running
fail: node-exporter: runsv not running
fail: postgres-exporter: runsv not running
fail: postgresql: runsv not running
fail: prometheus: runsv not running
fail: redis: runsv not running
fail: redis-exporter: runsv not running
fail: sidekiq: runsv not running
fail: unicorn: runsv not running

Version information:
apt-cache policy gitlab-ce
gitlab-ce:
Installed: 11.10.8-ce.0
Candidate: 11.10.8-ce.0

I do have a VM backup, so I can revert to 14.04 in order to do things differently if needed.

However, I was wondering if there are any suggestions on how to get GitLab running again as of the current state.

Update:
If I run /opt/gitlab/embedded/bin/runsvdir-start, GitLab starts and works fine. Hence, I suppose it`s something related to the service which is broken.

Any hints on how to proceed?

To be honest it’s going to be difficult. If the minimal services don’t start, eg postgres + one or two others, an upgrade cannot be performed. Had this been the case, then you could have just gotten gitlab-ce-11.10.8 for Ubuntu 18.04 or a later version from the gitlab upgrade path and then ran this to upgrade your existing install.

What you could try is to create a file in /etc/gitlab called skip-auto-reconfigure this will stop it attempting to do db migrations. Providing that you download gitlab-ce 11.10.8 for Ubuntu 18.04 this won’t be an issue anyway - it must be the same version as your currently installed for this to work successfully since we don’t want to skip db migrations if you download a newer version than what is installed. The skip-auto-reconfigure should just install the gitlab package. Once the package is installed, theoretically your services should all then start fine. Since you’ve already started gitlab-runsvdir, you might need to completely reboot the server and ensure with gitlab-ctl status that all services are running.

Once it’s sane for the version you have installed, you can then remove the skip-auto-reconfigure file, and then think about upgrading to newer versions but make sure to follow the upgrade path. Also, even if not thinking of upgrading, remove that file, else if you forget about it, you can have future problems when reconfiguring your server, or attempting upgrades. You can even at this point run an gitlab-ctl reconfigure to make sure it goes through all its steps and works fine, once the file has been removed.

Thank you very much, I now have GitLab running.

I first created a file in /etc/gitlab called skip-auto-reconfigure per your recommendation.

I then performed an installation of GitLab 11.11.7:

curl -s https://packages.gitlab.com/install/repositories/gitlab/gitlab-ce/script.deb.sh | sudo bash

sudo apt-get install gitlab-ce=11.11.7-ce.0

Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be upgraded:
  gitlab-ce
1 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 669 MB of archives.
After this operation, 64.4 MB of additional disk space will be used.
Get:1 https://packages.gitlab.com/gitlab/gitlab-ce/ubuntu bionic/main amd64 gitlab-ce amd64 11.11.7-ce.0 [669 MB]
Fetched 669 MB in 33s (20.5 MB/s)
(Reading database ... 154483 files and directories currently installed.)
Preparing to unpack .../gitlab-ce_11.11.7-ce.0_amd64.deb ...
gitlab preinstall:
gitlab preinstall: This node does not appear to be running a database
gitlab preinstall: Skipping version check, if you think this is an error exit now
gitlab preinstall:
Unpacking gitlab-ce (11.11.7-ce.0) over (11.10.8-ce.0) ...
Setting up gitlab-ce (11.11.7-ce.0) ...

gitlab: GitLab now ships with a newer version of PostgreSQL (10.7), but it is not yet
gitlab: enabled by default. To upgrade, RUN THE FOLLOWING COMMAND:

sudo gitlab-ctl pg-upgrade

gitlab: Note: This command does not support Geo instances yet. So we don't
gitlab: recommend running this command on Geo nodes. It will be supported
gitlab: in GitLab 12.0.

gitlab: For more details, please see:
gitlab: https://docs.gitlab.com/omnibus/settings/database.html#upgrade-packaged-postgresql-server
gitlab:
It looks like GitLab has not been configured yet; skipping the upgrade script.

       *.                  *.
      ***                 ***
     *****               *****
    .******             *******
    ********            ********
   ,,,,,,,,,***********,,,,,,,,,
  ,,,,,,,,,,,*********,,,,,,,,,,,
  .,,,,,,,,,,,*******,,,,,,,,,,,,
      ,,,,,,,,,*****,,,,,,,,,.
         ,,,,,,,****,,,,,,
            .,,,***,,,,
                ,*,.



     _______ __  __          __
    / ____(_) /_/ /   ____ _/ /_
   / / __/ / __/ /   / __ `/ __ \
  / /_/ / / /_/ /___/ /_/ / /_/ /
  \____/_/\__/_____/\__,_/_.___/


Thank you for installing GitLab!
GitLab should be available at https://git.crayon.com

For a comprehensive list of configuration options please see the Omnibus GitLab readme
https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/README.md

I then performed a reboot and checked that gitlab-ctl status was still showing all services as not running.

Next, I removed the file skip-auto-reconfigure from /etc/gitlab and ran:

gitlab-ctl reconfigure
*output truncated as it was huge*
Running handlers complete
Chef Client finished, 33/667 resources updated in 02 minutes 20 seconds
gitlab Reconfigured!

After that, all services is running:

gitlab-ctl status
run: alertmanager: (pid 3228) 159s; run: log: (pid 3155) 160s
run: gitaly: (pid 3844) 35s; run: log: (pid 3157) 160s
run: gitlab-monitor: (pid 3826) 36s; run: log: (pid 3160) 160s
run: gitlab-workhorse: (pid 3812) 37s; run: log: (pid 3167) 160s
run: logrotate: (pid 3230) 159s; run: log: (pid 3163) 160s
run: nginx: (pid 3168) 160s; run: log: (pid 3158) 160s
run: node-exporter: (pid 3657) 65s; run: log: (pid 3165) 160s
run: postgres-exporter: (pid 3227) 159s; run: log: (pid 3156) 160s
run: postgresql: (pid 3229) 159s; run: log: (pid 3161) 160s
run: prometheus: (pid 3192) 159s; run: log: (pid 3152) 160s
run: redis: (pid 3191) 159s; run: log: (pid 3153) 160s
run: redis-exporter: (pid 3224) 159s; run: log: (pid 3159) 160s
run: sidekiq: (pid 3189) 159s; run: log: (pid 3154) 160s
run: unicorn: (pid 3200) 159s; run: log: (pid 3164) 160s

Would you say the system is in a healthy state now? E.g. is it safe to proceed upgrading GitLab to newer versions?

Yes all services are running so all looks good. Check the gitlab upgrade docs for the upgrade path to follow as you will need to go through a few versions to get to the latest. You cannot do 11.x to 14.x as it will break.

Also after each upgrade you will need to ensure baclground migrations have finished. If unsure I will add step by step instructions when I get on my computer later as writing on phone not so easy.

Thanks again @iwalker for your awesome assistance on this issue. I`ll let you know should I need any further questions.

As per this link:

Your upgrade path is this: Upgrading GitLab | GitLab

11.11.8 -> 12.0.12 -> 12.1.17 -> 12.10.14 -> 13.0.14 -> 13.1.11 -> 13.8.8 -> 13.12.15 -> latest 14.0.Z -> latest 14.1.Z -> latest 14.Y.Z

the command to do that is this:

apt-get install gitlab-ce=11.11.8-ce.0

for example. You can get the exact version name from the package listing, so here for example: gitlab/gitlab-ce - Results for 'gitlab-ce_11.11.8' in gitlab/gitlab-ce

that is filtered for the 11.11.8 version. Mostly it should always finish with -ce.0 but might vary (I kind of remember writing once or twice -ce.1 but I could be wrong).

Also, once that apt upgrade command has finished, you can check background migrations either in the web interface, or run the following commands from this link: Upgrading GitLab | GitLab

gitlab-rails runner -e production 'puts Gitlab::BackgroundMigration.remaining'
gitlab-rails runner -e production 'puts Gitlab::BackgroundMigration.pending'

use sudo on all of those commands if you aren’t doing them as the root user. Once background migrations has completed (showing zero), you can then do the next upgrade so the apt-get upgrade command with the specific version number from the next gitlab version on the upgrade path.