Accidentally downgraded from 13.1 to 11.5, now offline and stuck

In my haste I accidentally downgraded my gitlab-ee omnibus installation from 13.1.11 to 11.5.11 (instead of up to 13.5) via a sudo apt upgrade gitlab-ee=11.5.11-ee.0. It fails during reconfigure at bash[migrate gitlab-rails database], what seems to be postgresql version incompatibilities. I cannot re-upgrade to 13.1.11 (preinstall prevents skipping a major version) and I cannot follow the upgrade path by upgrading to 11.11.8 (preinstall fails because reconfigure wasn’t successful).

The installation of 11.5 didn’t create a backup for an unknown reason. I do have a backup from version 13.0.14 when I upgraded from 13.0.14 to 13.1.11 approximately 20 hours ago.

What is the right course of action here?

First, you will need that gitlab backup from 13.0.14, then the /etc/gitlab/gitlab.rb and the secrets.json file also within this directory.

Then you will need to remove gitlab from your server to then install 13.0.14 to restore your backup. Place git gitlab.rb and secrets.json back in /etc/gitlab after installing the 13.0.14 package and run gitlab-ctl reconfigure. Once this has been done, you can move the gitlab backup into /var/opt/gitlab/backups and run the restore procedures based on the gitlab documentation.

Once this is restored, check you can access everything, and then do your upgrade to 13.5. You don’t need to normally put apt upgrade gitlab-ee=xx.x.xx-ee.0 - you can just do apt-upgrade and it will find it. You would only need to do such a thing when upgrading from a really old release that needs you to follow the upgrade path in the Gitlab upgrade docs.

Please read restore docs: Back up and restore GitLab | GitLab

1 Like

I was hesitant to restore from the backup because the backup was a bit outdated. Here’s what I ended up doing to get back to 13.11.1, which seems to have worked properly. These steps probably won’t work for every accidental downgrade, but my downgrade here never ran any DB migrations which may have helped out.

  1. Disable automatic DB migration by setting gitlab_rails['auto_migrate'] = false.
  2. Re-run sudo gitlab-ctl reconfigure, which now completes successfully.
  3. Disable auto-reconfigure: sudo touch /etc/gitlab/skip-auto-reconfigure.
  4. Upgrade to 11.11.8, 12.0.12, 12.10.14, 13.0.14, 13.1.11.
  5. Re-enable automatic DB migration.
  6. Clear the auto-reconfigure disable: sudo rm /etc/gitlab/skip-auto-reconfigure.
  7. Run sudo gitlab-ctl reconfigure.
3 Likes