Gitlab Self Managed Instance Upgrade Issues (Dropdown Menus Not working)

Hello Everyone,

We have recently migrated our gitlab self-managed instance to a new server and then upgraded the gitlab version to that latest version 13.8.2 using the recommended upgrade path below.

upgrade path:
From: source version → 12.7.2-ee.0
intermediate version → 12.10.14-ee.0
intermediate version → 13.0.14-ee.0
To: final/target version → 13.8.2

Since the upgrade, all the dropdown menus stopped working.

I have found an old forum below that discussed about the same issue. But there was no clear resolution.

Have anyone of you come across this issue during the upgrade and if so, were you able to resolve it? Please Advise. Thank you for your support.

Thanks,
akaiserk

I’m guessing that your problem is due to a missed upgrade release:

https://docs.gitlab.com/ee/update/

in particular from the upgrade path:

8.11.x → 8.12.0 → 8.17.7 → 9.5.10 → 10.8.7 → 11.11.8 → 12.0.12 → 12.1.17 → 12.10.14 → 13.0.14 → 13.1.11 - > 13.x (latest)

my emphasis in bold for the part relative to your upgrade - you seem to have missed 13.1.11 when going from 13.0.14.

If that’s the case, I would restore my server and start again the upgrade process. Also make sure as per the upgrade docs to ensure all background migrations have finished after each upgrade as that can also cause you problems.

Hi, Thank you so much for your response.

We currently have 2 different versions of Gitlab running in our Dev and Production Environments.

For Production, Gitlab Support has recommended the following upgrade path
12.10.6 → 12.10.14 → 13.0.14 → 13.7.2

For Dev, We applied the same logic and used the following upgrade path.
12.7.2 → 12.10.14 → 13.0.14 → current version

Any particular reason to include 13.1.11 in the upgrade path?

Best Regards,
akaiserk

Well the Gitlab upgrade documentation suggests that upgrade path with missing version. Check the link in my previous post. If Gitlab support suggested it then I guess that means you have paid support with them. In that case you may want to contact them directly as this forum is mostly for free support for people without subscriptions.

I helped someone upgrade recently and they followed the path as suggested in the docs and it worked. They also checked after each upgrade that the background migrations had finished (which is also in the upgrade doc link). So I expect that either the missed upgrade or continuing upgrades when background migrations are still running is the source of the problem.

Thank you so much for your recent response.

I am fairly new to gitlab and trying to get as much help as possible from different sources. I have read the documentation a little bit and still confused with the whole concept of background migrations.

I have a few basic questions about background migrations. What exactly are background migrations? Do background migrations happen during the upgrade or after the upgrade process automatically? Are the background migrations part of the omnibus package? Can we control on how and when the background migrations happen?

I would really appreciate if you could provide some context around the concept of background migrations.

Thank you.

Hi,

The background migrations are tasks most likely happening on the database, or things going on in the sidekiq queue (not entirely sure). But either way, after an upgrade, you need to check it by running the following command (it’s in the upgrade docs):

gitlab-rails runner -e production 'puts Gitlab::BackgroundMigration.remaining'

now if that reports zero migrations, that means they have finished, and you can continue the next upgrade. And then repeat the migration command, and so on until finally at the latest version.

Hi,

It looks like this was the reason why I was having strange issues. I wasn’t doing the check for background migrations and waiting for them to complete before upgrading to the next version in the upgrade path. After adding this step, I don’t see the issue with the Dropdown menus any more. I am currently in the last stretch now upgrading to the latest version. Thank you for your prompt responses and support. I really appreciate it.