Gitlab background migrations

Hi all,

Could you please suggest me from which version, the background migrations are needed ?

Effectively, all versions, any upgrade that you can do requires checking background migrations to finish before continuing to the next one.

Sidekiq::Queue.new(‘background_migration’).size

can we also do background checks using the above command on gitlab-rails console ? Also on the Gui background migrations if the queue is clear or 0 then should also be okay ?

You can check background migrations in the web UI, and also using the gitlab-rails command. If it shows zero in either or both places, then upgrade can be continued. If not, then need to wait until they have finished. Somewhere in Gitlab docs it also tells you how to get migrations to finish if they are stuck. Here for more info: Maintenance Rake tasks | GitLab

Sidekiq::Queue.new(‘background_migration’).size --------- is this correct to check background migrations ?

No, as per the Gitlab upgrade docs under background migrations section:

gitlab-rails runner -e production 'puts Gitlab::BackgroundMigration.remaining'

undefined method `remaining’ for Gitlab::BackgroundMigration:Module

The given command works fine here, so I suspect there might be a serious problem with your installation somewhere, also because I just read another thread here where @iwalker tries to help you:

So take care and backups.

In our internal documentation the variable you’re referring to is still listed as the way to check the queue before the command @iwalker quoted came along (our documentation says it was in 12.9 from March 2020). It should be safe to try (but as your installation seems to have some problem, there’s no guarantee) and here it is only about one second slower, but the commands take above 40 seconds, so that doesn’t seem significant.

Hi I have not started the upgrade yet. I am currently in the version 11.11.5 and wanted to proceed for upgradation in the coming days and when i ran background check i got the above error… What you mean by serious problem with installation ?

i ran ```
gitlab-rake gitlab:check

Also `gitlab-rake db:migrate:status` shows the status "up" what else could be a serious problem

That’s the thing, I have no idea, but both the error you quote in this thread and the one you quoted in the other thread indicates that something is wrong.

If you can get an upgrade to run, you can hope that fixes everything.

You can hope that someone comes by that can tell what might have caused that error and knows how to fix it.

You can make a support request, quoting your errors, but as you’re on CE, it might take some time (days, probably) before someone gets back to you, and I don’t know if the answer will be usable.

If you have (or can make) usable backups, you can wipe the installation. install GitLab again and restore a backup.

Note that there’s a risk any command (including gitlab-backup create) performed against a misbehaving system (as yours seems to be at the moment) will fail and cause even bigger problems.

But now I just realised that you’ve said you’re on 11.11.5, that’s a lot earlier than my ressources say that the command @iwalker gave in this thread is supposed to work, so that failure might not be serious (I don’t have a GitLab-server with that old a version running, so I can’t see what the expected error should be), and the command giving problems in the other thread is basically the same, so we actually don’t have any indication that anything is wrong.

Try checking the variable you mentioned yourself, a complete command to do so is:

sudo gitlab-rails runner 'puts Sidekiq::Queue.new("background_migration").size'

If that says 0, you should be able to continue upgrading. Remember to check (and wait) for this between each step in the upgrade procedure. When you pass 12.9 (i.e. have gone to 12.10.14) the other command for checking the queue size should start working, it will probably be good to test.

Yep, that’ll explain it so the install could be fine, just wrong command. There’s no gitlab docs online for this, but it should be accessible via docker from the archive to run locally. I’ll see if I can check that locally now and get an answer for the correct command.

Hi Grove, yes thats what I am quoting from the beginning because i see the background migrations start after 12.0 version but not with 11 version.

As mentioned

Sidekiq::Queue.new(‘background_migration’).size it gives me the output “0” and i think i am good to go to proceed for upgradation.

Well I downloaded the archive docs and there is nothing in the upgrade guide which says how to check migrations. Effectively all you do is run two things when updating at that point:

gitlab-ctl reconfigure

which would then run DB migrations after installing the upgrade package. After that you run post-deployment DB migrations with:

gitlab-rake db:migrate

unfortunately the docs are not searchable, as the docker images is constantly referring to docs.gitlab.com instead of providing search results from the docker instance itself. So I’ve had to go manually through the docs as much as possible to find relative stuff.

As long back as I’ve been responsible for keeping GitLab upgraded on our servers (that was around 10.6/10.7), the documentation has said to make sure background migrations were finished, it was just hard to find out how to actually do that. So background migrations don’t start after 12.0, when you’re on 11.11, there might not be any until you get to 12.0, but that’s mostly a coincidence and no reason not to perform this check until you get there.

@grove yep, that’s what I found. I ran from docker the docs for 11.11.10 and pretty much read every page that I could think of where such a command would show up. Sadly didn’t manage to find anything.

hi @iwalker @grove : thanks for the replies…so i think i can skip checking migration for this version.

You should never skip checking for background migrations!

As long as you’re on a version prior to 12.9 you’ll need to use the

sudo gitlab-rails runner 'puts Sidekiq::Queue.new("background_migration").size'

command to do so. When you get to versions past 12.9, you can choose between that command and the (in my experience - very - slightly faster) command:

sudo gitlab-rails runner -e production 'puts Gitlab::BackgroundMigration.remaining'
```

@grove : thank you for the reply and suggestions