Upgrade Problems Debian 14.2.4

Hey,

I have a problem after migration. I called apt-get upgrade and gitlab-ee startet to upgrade to 14.2.4 (from 14.0.1).

Now I am getting the following message:

STDOUT: rake aborted!
StandardError: An error has occurred, all later migrations canceled:

Expected batched background migration for the given configuration to be marked as 'finished', but it is 'active':	{:job_class_name=>"CopyColumnUsingBackgroundMigrationJob", :table_name=>"ci_job_artifacts", :column_name=>"id", :job_arguments=>[["id", "job_id"], ["id_convert_to_bigint", "job_id_convert_to_bigint"]]}

Finalize it manualy by running

	sudo gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,ci_job_artifacts,id,'[["id"\, "job_id"]\, ["id_convert_to_bigint"\, "job_id_convert_to_bigint"]]']

For more information, check the documentation

	https://docs.gitlab.com/ee/user/admin_area/monitoring/background_migrations.html#database-migrations-failing-because-of-batched-background-migration-not-finished
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:1129:in `ensure_batched_background_migration_is_finished'
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20210706212710_finalize_ci_job_artifacts_bigint_conversion.rb:14:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:61:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'

Caused by:
Expected batched background migration for the given configuration to be marked as 'finished', but it is 'active':	{:job_class_name=>"CopyColumnUsingBackgroundMigrationJob", :table_name=>"ci_job_artifacts", :column_name=>"id", :job_arguments=>[["id", "job_id"], ["id_convert_to_bigint", "job_id_convert_to_bigint"]]}

Finalize it manualy by running

	sudo gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,ci_job_artifacts,id,'[["id"\, "job_id"]\, ["id_convert_to_bigint"\, "job_id_convert_to_bigint"]]']

For more information, check the documentation

	https://docs.gitlab.com/ee/user/admin_area/monitoring/background_migrations.html#database-migrations-failing-because-of-batched-background-migration-not-finished
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:1129:in `ensure_batched_background_migration_is_finished'
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20210706212710_finalize_ci_job_artifacts_bigint_conversion.rb:14:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:61:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => db:migrate
(See full trace by running task with --trace)
== 20210706212710 FinalizeCiJobArtifactsBigintConversion: migrating ===========

I visited the Batched background migrations | GitLab and tried to downgrade by apt-get but this fails.

What can I do now?

Bests
Welano

Hi, i had the same problem.
I am using GitLab EE with the offical Dockerhub image.

I upgraded from 13.11.3-ee to 13.12.12,then to 14.0.10.
The upgrade from 14.0.10 got 14.2.4 threw the following error:

gitlab    | rails_migration[gitlab-rails] (gitlab::database_migrations line 51) had an error: Mixlib::ShellOut::ShellCommandFailed: bash[migrate gitlab-rails database] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb line 16) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
gitlab    | ---- Begin output of "bash"  "/tmp/chef-script20210922-31-8y7b80" ----
gitlab    | STDOUT: rake aborted!
gitlab    | StandardError: An error has occurred, all later migrations canceled:
gitlab    | 
gitlab    | Expected batched background migration for the given configuration to be marked as 'finished', but it is 'active':	{:job_class_name=>"CopyColumnUsingBackgroundMigrationJob", :table_name=>"ci_job_artifacts", :column_name=>"id", :job_arguments=>[["id", "job_id"], ["id_convert_to_bigint", "job_id_convert_to_bigint"]]}
gitlab    | 
gitlab    | Finalize it manualy by running
gitlab    | 
gitlab    | 	sudo gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,ci_job_artifacts,id,'[["id"\, "job_id"]\, ["id_convert_to_bigint"\, "job_id_convert_to_bigint"]]']
gitlab    | 
gitlab    | For more information, check the documentation
gitlab    | 
gitlab    | 	https://docs.gitlab.com/ee/user/admin_area/monitoring/background_migrations.html#database-migrations-failing-because-of-batched-background-migration-not-finished
gitlab    | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:1129:in `ensure_batched_background_migration_is_finished'
gitlab    | /opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20210706212710_finalize_ci_job_artifacts_bigint_conversion.rb:14:in `up'
gitlab    | /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:61:in `block (3 levels) in <top (required)>'
gitlab    | /opt/gitlab/embedded/bin/bundle:23:in `load'
gitlab    | /opt/gitlab/embedded/bin/bundle:23:in `<main>'
gitlab    | 
gitlab    | Caused by:
gitlab    | Expected batched background migration for the given configuration to be marked as 'finished', but it is 'active':	{:job_class_name=>"CopyColumnUsingBackgroundMigrationJob", :table_name=>"ci_job_artifacts", :column_name=>"id", :job_arguments=>[["id", "job_id"], ["id_convert_to_bigint", "job_id_convert_to_bigint"]]}
...
...
gitlab    |     System Info:
gitlab    |     ------------
gitlab    |     chef_version=15.14.0
gitlab    |     platform=ubuntu
gitlab    |     platform_version=20.04
gitlab    |     ruby=ruby 2.7.2p137 (2020-10-01 revision 5445e04352) [x86_64-linux]
gitlab    |     program_name=/opt/gitlab/embedded/bin/chef-client
gitlab    |     executable=/opt/gitlab/embedded/bin/chef-client
gitlab    |     
gitlab    | 
gitlab    | Running handlers:
gitlab    | Running handlers complete
gitlab    | Chef Infra Client failed. 11 resources updated in 28 seconds

I checked for background migrations before the upgrade, which returned 0:
gitlab-rails runner -e production ‘puts Gitlab::BackgroundMigration.remaining’

The docker container with gitlab keeps terminating and restarting, but i could successfully
run the command stated in the error message:
gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,ci_job_artifacts,id,’[[“id”, “job_id”], [“id_convert_to_bigint”, “job_id_convert_to_bigint”]]’]

But it kept restarting and logging the same error.
I tried again to run the migration, it said it was successful, but the same error reappeared.

I successfully downgraded to 4.1.0, because of the notes in the upgrading doc, which works. I did another attempt from 4.1.0 to 4.2.4, but it failed with the same error.
Then i downgraded again to 4.1.0, let the migration run
gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,ci_job_artifacts,id,’[[“id”, “job_id”], [“id_convert_to_bigint”, “job_id_convert_to_bigint”]]’]
upgraded to 4.1.5,
upgraded to 4.2.0,
upgraded to 4.2.4 successfully.

The Upgrade Path from the docs (latest 13.12.Z → latest 14.0.Z → latest 14.Y.Z) does not seem to be accurate.

Hi @welano - did you follow our downgrade docs? There is a very specific order of commands you need to follow - additionally you will need a backup from 14.0.1 as part of the downgrade, so you can restore that. Downgrading via apt-get alone does not downgrade the database which is why the restore step is important.

Once you are on 14.0.1, you will need to upgrade to the latest 14.0.x release (which is 14.0.10 at the time of writing). Then, follow the link you shared Batched background migrations | GitLab and check the status of the batched background migration - once that is done it should be safe to update to 14.2 or later.

Here are some handy links for future reference:

Hi @gork74 - this is definitely odd. Based on your post, I would agree you have followed our recommendations as written in our docs. Starting on 14.0.10 and checking that Batched background migrations | GitLab was zero before proceeding to the latest 14.Y.Z should have worked.

Based on the rest of your post, it seems you were able to get upgraded to 14.2.Z successfully which is awesome!

The Upgrade Path from the docs (latest 13.12.Z → latest 14.0.Z → latest 14.Y.Z) does not seem to be accurate.

I did go and consult with my team and we are confident that the current upgrade path is correct latest 14.0.Z → latest 14.Y.Z. I’m not aware of any other users that needed to upgrade to 14.1.Z as an intermediary step but we will definitely keep an eye out. Thanks for posting your experiences though!

FYI: I just ran into the same error on Ubuntu 18, coming from 13.8.8 => 13.12.10 => 14.0.10 => 14.3.0. The last step from 14.0.10 to 14.3.0 failed as described above. Running omnibus but not inside docker.

We are facing the same problem. Upgrade from 13.12.10 to 14.0.0 worked fine, the following upgrade to the latest 14.3.x failed

Running handlers:
There was an error running gitlab-ctl reconfigure:

rails_migration[gitlab-rails] (gitlab::database_migrations line 51) had an error: Mixlib::ShellOut::ShellCommandFailed: bash[migrate gitlab-rails database] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/res
ources/rails_migration.rb line 16) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of "bash"  "/tmp/chef-script20210926-25225-1b1ec3z" ----
STDOUT: == 20210317210338 AddValidRunnerRegistrars: migrating =========================
-- add_column(:application_settings, :valid_runner_registrars, :string, {:array=>true, :default=>["project", "group"]})
   -> 0.0201s
== 20210317210338 AddValidRunnerRegistrars: migrated (0.0202s) ================

...


Expected batched background migration for the given configuration to be marked as 'finished', but it is 'active':	{:job_class_name=>"CopyColumnUsingBackgroundMigrationJob", :table_name=>"events", :column_name
=>"id", :job_arguments=>[["id"], ["id_convert_to_bigint"]]}

Finalize it manualy by running

	sudo gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,events,id,'[["id"]\, ["id_convert_to_bigint"]]']

For more information, check the documentation

Our installation was broken after that and we got a 500 SERVER ERROR after the login into Gitlab. I needed to downgrade the package to 14.0.0 and install the postgresql backup.

Would be great if someone could have a look at that, right now it seems not possible to upgrade.

I suggest doing this as your upgrade path:

  1. Upgrade to latest 14.0.x release
  2. Upgrade to latest 14.1.x release
  3. Upgrade to latest 14.2.x release
  4. Upgrade to 14.3.0.

Should work just make sure that background migrations finished before atarting the next upgrade.

Just ran into this problem during our primary server upgrade(omnibus). The standby server did not have this problem doing the “same” migration path, likely do to there being almost no data on the standby server or possibly the small 14.0.Z difference.
primary server: 13.12.12 → 14.0.10 → 14.2.4 # failed with this error (upgrade was 20210925)
standby server: 13.12.12 → 14.0.11 → 14.2.4 # had no issues (14.0.11 was released today 20210927)

These checks were run before each update:

  gitlab-rake db:migrate:status | grep -v '   up '
  gitlab-rails runner -e production 'puts Gitlab::BackgroundMigration.remaining'
  gitlab-rake gitlab:check SANITIZE=true

Same problem here 13.12.12 → 14.0.11 → 14.3.0 failed. Should I downgrade?
I am also omnibus installation (CE) on Debian Buster.

Same as @Snowmanko
i tried from 13.12.12 to 14.0.11. then 14.1.* (worked ok). from 14.1.* to 14.2.* it breaks.

with all the precautions mentioned above, obviously.

any updates or tips on how to get to latest version?

kinda stuck

1 Like

Hi, having the exact same issue as ionut.tanasa.commit and Snowmanko: breaks between 14.1.7-ee.0 and 14.2.5-ee.0

1 Like

I ran into this problem on centos. I am using the yum repository.
There seems to be a safe path going 14.1.7 → 14.2.1 → 14.2.5 → 14.3.1 (latest at the moment)
Going 14.1.7 → 14.2.5 (or 14.1.7 → 14.3.1 for that matter) fails.
Note that 14.2.0 has a problem with the rpm that makes it fail if you try using it for a yum update from earlier versions. That is the reason for using 14.2.1 in this chain
Whenever the updates failed I did a yum remove gitlab-ce and reinstalled the previous version. Then restored the database from the backup generated by the yum update.

1 Like

Following some further conversation I’ve had internally with our Database team, we have improved our documentation on how to recover from this error. The MR was merged 4 hours ago and is now live on our docs website.

Please see the following updated links:

If you have any questions please let me know.

I had the issue with docker on the ee - version. Is this fixed as well?

Yes, the same instructions should work on all distributions of GitLab including Docker.

The problem occurs when batched migrations have not completed before upgrading from 14.0.5 - 14.0.Z or 14.1.Z to a higher version. As long as you have checked the status of the batched migrations and confirmed that they are all marked as finished before you upgrade to 14.2 and beyond, then the upgrade should be successful.

If the upgrade isn’t successful, the docs I mentioned previously (Batched background migrations | GitLab) will contain the next steps you can try.

1 Like

My point was: the update on docker runs by itself without me being able to interfere or try anything. Thats an automated process afaik.

i can confirm that this works. i was on 14.2.0 and went 14.2.1 → 14.2.5 → 14.3.1

ty very much sir @b1r63r

Hi, I can also confirm that this path works for the gitlab/gitlab-ee docker release. I updated from 14.3.1 to 14.3.2 afterwards. That also worked fine.

Upgrading from source here:
13.12.12 → 14.0-stable
14-0-stable → 14-1-stable
14-1-stable → 14-2-stable → Background migration error suggests the Omnibus only fix:

sudo gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,ci_build_needs,id,'[["build_id"]\, ["build_id_convert_to_bigint"]]']

I’ve found the process to manually finish the background migrations for Omnibus installation (Batched background migrations | GitLab), but I don’t appear to find a work-around if upgrading from source.

On this page (Upgrading GitLab | GitLab) running:

sudo -u git -H bundle exec rails runner -e production 'puts Gitlab::BackgroundMigration.remaining'

shows 8 Background migrations remaining.

However, attempting to launch the rails console:

sudo -u git -H bundle exec rails RAILS_ENV=production

Results in:

Running the full GitLab test suite takes significant time to pass. We recommend using one of the following spec tasks: Learn more at https://docs.gitlab.com/ee/development/rake_tasks.html#run-tests.

Which is a page only for Omnibus.

Anyone know how I can finish the background migrations from source install of GitLab?