Inconsistent Migrations from v16.1.5 to v16.3.6 Upgrade

Hello Team,

We use the Omnibus tool to maintain and upgrade our Gitlab Setup.

On upgrading from v16.1.5 to v16.3.6 in one setup, the Background Migrations went through automatically and showed up in the Admin Panel UI.

Whereas in another setup running the exact same command for upgrade didn’t run the Background Migrations when upgraded to v16.3.6 and the Migrations had to be manually triggered using gitlab-rake db:migrate

Does anyone know why this happens and is it a good practice to run gitlab-rake db:migrate:status before and after every upgrade?

Thanks

From this page: Zero downtime upgrades | GitLab in point C:

perhaps you have that configured in /etc/gitlab/gitlab.rb

Hey, thanks for the quick reply!

The above configuration has been disabled in both of the platform gitlab.rb files. So they should ideally show similar behaviour.

Is this just a thing where sometimes the migrations do not run post the upgrade and we need to make sure they run after the upgrade?

For me, migrations have always ran. I do now that some things can also be skipped by creating certain files in /etc/gitlab but I think that’s mostly to do with skipping reconfigure, I don’t remember anything for stopping migrations that way. But you can check for the existence of files in /etc/gitlab that may be causing this issue.

I’ve also never had that option enabled in my Gitlab config, even by default it’s always enabled for migrations to run. Migrations usually get queued and show in the web interface, so it’s weird that yours haven’t been running on one of your servers if files or config options haven’t been set.

I’d always recommend running gitlab-rake db:migrate:status simply because it can be finicky. I’ve noticed it way more on versions <13 but as you can see- sometimes even on the newer versions it might not run for whatever reason.

Yes I’ll check that, thanks for the help!

Yes we now run gitlab-rake gitlab:check SANITIZE= true &gitlab-rake db:migrate:status before and after the upgrade just to be sure.

Thanks for the reply!