GitLab EE not more possible to Update after enable ESM

I have enabled UA Infra: Extended Security Maintenance (ESM). Now after enter apt-get update, gitlab will not more update.

In my administration account i can see update are available.

Auswahl_001

Hi @Joulindo
please specify whats the Ubuntu version.

Ubuntu 16.04.7 LTS (GNU/Linux 4.4.0-213-generic x86_64)

Latest version of GitLab Omnibus package for Ubuntu 16 (Xenial) is 13.12.9 released 10 hours ago. At the time of your post the latest version of GitLab Omnibus package for Ubuntu Xenial was 13.12.8.

Ubuntu Xenial is not supported anymore for GitLab 14.x. See docs.

The GitLab UI shows update available, but that does not take any prerequisites into consideration.

I need Upgrade to Ubuntu 18 or GitLab 15?

You need to upgrade to Ubuntu 18 or 20.

Hi @Joulindo, welcome to the GitLab Community forum!

As others have noted, GitLab stopped building and releasing new packages for Ubuntu 16.04 prior to the 14.x release.

Here are your upgrade options:

Upgrade Options

Option A: Migrate GitLab from Ubuntu 16.04 server new Ubuntu 18/20.04 server

  1. Backup GitLab on 16.04 server
  2. Spin up new Ubuntu 18.04 or 20.04 server on seperate box
  3. Install GitLab on 18/20.04 box using same GitLab version installed on Ubuntu 16.04 box
  4. Copy backup to new 18/20.04 server
  5. Restore backup on new 18/20.04 server

Option B: In-place OS upgrade

  1. Upgrade to the latest version of available packages. OS upgrade will fail unless you’re running the latest version of 16.04 packages. sudo apt-get dist-upgrade
  2. Initiate OS upgrade: sudo do-release-upgrade (ensure you’ll be available and attentive, this process will require user interaction and shouldn’t be stopped once it’s started)
  3. Follow prompts, select to not change configuration if in doubt.
  4. Reboot
  5. Add GitLab package repository for your new OS version
curl https://packages.gitlab.com/install/repositories/gitlab/gitlab-ee/script.deb.sh | sudo bash
  1. sudo apt update

(good additional resource: How To Upgrade to Ubuntu 18.04 Bionic Beaver | DigitalOcean)

If you encounter any problems, there are some troubleshooting steps here.

2 Likes

@gitlab-greg Thanks for the thorough reply. A few follow-up questions

  1. Is either Option recommended over the other or should both work equally as well (in theory)?

  2. Is the answer basically the same for gitlab-ce (aside from a slightly different omnibus URL in step #5)?

  3. In Option B will step #5 just update the existing apt repository to reference bionic or focal instead of xenial?

Hi @fleish :wave: , welcome to the GitLab Community forum! :tada:

  1. Is either Option recommended over the other or should both work equally as well (in theory)?

I’ve tested both options extensively and in my experience they both work equally well. Option A is zero-risk, Option B carries some risk.

I suggest migration (Option A) instead of in-place upgrade (Option B) because it mitigates the risk of data loss or downtime. In-place upgrades work great 99% of the time, but there’s a slight chance that if system crashes during the OS upgrade it could block SSH connections and/or leave the system in a non-bootable state. Recovery from this state without a filesystem snapshot could require physical (Keyboard + Monitor) access to the server - so unless you’re running GitLab on an on-prem server, an in-place upgrade carries above-zero chance of “bricking” your machine.

  1. Is the answer basically the same for gitlab-ce (aside from a slightly different omnibus URL in step #5)?

Exactly! If you’re running gitlab-ce instead of gitlab-ee, simply replace -ee with -ce in these examples.

  1. In Option B will step #5 just update the existing apt repository to reference bionic or focal instead of xenial?

That’s correct! With Option B (in-place OS upgrade), the xenial repository sources get commented-out in /etc/apt/sources.list.d/. Until the apt repository source for gitlab-[c,e]e is updated to reference bionic or focal, no upgraded versions will show as available via apt package manager. The one-liner in Option B step 5 adds appropriate repository source your system’s apt sources.

Alternatively, one could un-comment the xenial repository source in your /etc/apt/sources.list.d/gitlab_gitlab-[c,e]e.list file and change xenial to bionic or focal following the upgrade, then run apt update to have apt fetch the latest package versions.

Thanks for the welcome, @gitlab-greg!

I went ahead with Option B and wanted to note a few things.

First is after apt update in step #6 it is also necessary to run apt upgrade additional wrinkle.

Another is gitlab could not upgrade itself from version 13.12.10 to the latest version 14.2.3 directly. I had to first do an intermediate install to the latest 14.0.x version. From there I was able to goto 14.2.3, although that is not specifically covered in Upgrading GitLab | GitLab

I also ran into an issue where the DB updates failed somewhere and I had to run a command manually to get them to complete and then re-run apt upgrade for things to finish up.

All in all, things seem to have gone fairly smoothly other than those items. Hopefully others will see this and be prepared for those possible exceptions.

1 Like

@fleish yes it is configured in the upgrade guide: Upgrading GitLab | GitLab

 latest 13.12.Z -> latest 14.0.Z -> 14.1.Z -> latest 14.Y.Z

it shows exactly the upgrade path to go from 13.12.10 to latest 14.0.x latest 14.1.x and then to 14.2.x You missed a couple of in-between upgrades, so I hope you won’t have any issues due to potentially missed migrations.

I get error after upgrade

Hit:5 https://packages.gitlab.com/gitlab/gitlab-ee/ubuntu bionic InRelease
Fetched 14.9 kB in 1s (17.1 kB/s)
Reading package lists… Done
Reading package lists… Done
Building dependency tree
Reading state information… Done
Calculating upgrade… Done
The following packages will be upgraded:
gitlab-ee
1 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 1,044 MB of archives.
After this operation, 205 MB of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 https://packages.gitlab.com/gitlab/gitlab-ee/ubuntu bionic/main amd64 gitlab-ee amd64 14.6.0-ee.0 [1,044 MB]
Fetched 1,044 MB in 19s (56.1 MB/s)
(Reading database … 189243 files and directories currently installed.)
Preparing to unpack …/gitlab-ee_14.6.0-ee.0_amd64.deb …
gitlab preinstall: It seems you are upgrading from major version 13 to major version 14.
gitlab preinstall: It is required to upgrade to the latest 14.0.x version first before proceeding.
gitlab preinstall: Please follow the upgrade documentation at Upgrading GitLab | GitLab
dpkg: error processing archive /var/cache/apt/archives/gitlab-ee_14.6.0-ee.0_amd64.deb (–unpack):
new gitlab-ee package pre-installation script subprocess returned error exit status 1
Errors were encountered while processing:
/var/cache/apt/archives/gitlab-ee_14.6.0-ee.0_amd64.deb
E: Sub-process /usr/bin/dpkg returned an error code (1)

Hi @Joulindo :wave:

The error you’re seeing is actually unrelated to Ubuntu 16.04 ESM.

The cause of this error is embedded in the error message itself:

TL;DR: You can’t upgrade to 14.6 from your current 13.x version. You need to follow an upgrade path. Upgrading GitLab | GitLab and use version-specific upgrade instructions.

As of right now, the upgrade path for 13.0 to 14.6 will be:

13.1.11 → 13.8.8 → 13.12.12 → 14.0.12 → 14.1.8 → 14.6.0

Be sure to check for background migrations between upgrades . Some version upgrades, like 14.0, kick off background & database migrations that can take hours or even days to successfully complete. Upgrading before the migrations have completed can (and likely will) cause known problems.

What are the correct right command that i need do in terminal?

@Joulindo that will depend on your Linux Distribution and package manager (for Linux package installs), installation method (for Docker/Helm Charts), and whether you’re using GitLab CE or GitLab EE.

Command examples for upgrading to a specific version of GitLab via Linux package manager can be found here:

If you’re using Docker, you’ll use the instrucitons here GitLab Docker images | GitLab and modify the image tag from latest to the tag that corresponds with each version in your upgrade path.

was able to fix the problem with this:

Installing packages from: https://packages.gitlab.com

Install latest 13.12.x
gitlab-ctl reconfigure
sudo gitlab-ctl restart redis
reboot
install latest 14.1.x
gitlab-ctl reconfigure
sudo gitlab-ctl restart redis
reboot
install latest 14.2.x
gitlab-ctl reconfigure
sudo gitlab-ctl restart redis
reboot
install latest 14.3.x
gitlab-ctl reconfigure
sudo gitlab-ctl restart redis
reboot
install latest 14.4.x
gitlab-ctl reconfigure
sudo gitlab-ctl restart redis
reboot
install latest 14.5.x
gitlab-ctl reconfigure
sudo gitlab-ctl restart redis
reboot
wget --content-disposition https://packages.gitlab.com/gitlab/gitlab-ee/packages/ubuntu/focal/gitlab-ee_14.6.6-ee.0_amd64.deb/download.deb
gitlab-ctl reconfigure
sudo gitlab-ctl restart redis
reboot
install latest 14.7.x
gitlab-ctl reconfigure
sudo gitlab-ctl restart redis
reboot
install latest 14.8.x

gitlab-ctl reconfigure
sudo gitlab-ctl restart redis
reboot

1 Like