Upgrade Problems Debian 14.2.4

welano · September 21, 2021, 8:42am

Hey,

I have a problem after migration. I called apt-get upgrade and gitlab-ee startet to upgrade to 14.2.4 (from 14.0.1).

Now I am getting the following message:

STDOUT: rake aborted!
StandardError: An error has occurred, all later migrations canceled:

Expected batched background migration for the given configuration to be marked as 'finished', but it is 'active':	{:job_class_name=>"CopyColumnUsingBackgroundMigrationJob", :table_name=>"ci_job_artifacts", :column_name=>"id", :job_arguments=>[["id", "job_id"], ["id_convert_to_bigint", "job_id_convert_to_bigint"]]}

Finalize it manualy by running

	sudo gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,ci_job_artifacts,id,'[["id"\, "job_id"]\, ["id_convert_to_bigint"\, "job_id_convert_to_bigint"]]']

For more information, check the documentation

	https://docs.gitlab.com/ee/user/admin_area/monitoring/background_migrations.html#database-migrations-failing-because-of-batched-background-migration-not-finished
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:1129:in `ensure_batched_background_migration_is_finished'
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20210706212710_finalize_ci_job_artifacts_bigint_conversion.rb:14:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:61:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'

Caused by:
Expected batched background migration for the given configuration to be marked as 'finished', but it is 'active':	{:job_class_name=>"CopyColumnUsingBackgroundMigrationJob", :table_name=>"ci_job_artifacts", :column_name=>"id", :job_arguments=>[["id", "job_id"], ["id_convert_to_bigint", "job_id_convert_to_bigint"]]}

Finalize it manualy by running

	sudo gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,ci_job_artifacts,id,'[["id"\, "job_id"]\, ["id_convert_to_bigint"\, "job_id_convert_to_bigint"]]']

For more information, check the documentation

	https://docs.gitlab.com/ee/user/admin_area/monitoring/background_migrations.html#database-migrations-failing-because-of-batched-background-migration-not-finished
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:1129:in `ensure_batched_background_migration_is_finished'
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20210706212710_finalize_ci_job_artifacts_bigint_conversion.rb:14:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:61:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => db:migrate
(See full trace by running task with --trace)
== 20210706212710 FinalizeCiJobArtifactsBigintConversion: migrating ===========

I visited the Batched background migrations | GitLab and tried to downgrade by apt-get but this fails.

What can I do now?

Bests
Welano

gork74 · September 22, 2021, 12:01pm

Hi, i had the same problem.
I am using GitLab EE with the offical Dockerhub image.

I upgraded from 13.11.3-ee to 13.12.12,then to 14.0.10.
The upgrade from 14.0.10 got 14.2.4 threw the following error:

gitlab    | rails_migration[gitlab-rails] (gitlab::database_migrations line 51) had an error: Mixlib::ShellOut::ShellCommandFailed: bash[migrate gitlab-rails database] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb line 16) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
gitlab    | ---- Begin output of "bash"  "/tmp/chef-script20210922-31-8y7b80" ----
gitlab    | STDOUT: rake aborted!
gitlab    | StandardError: An error has occurred, all later migrations canceled:
gitlab    | 
gitlab    | Expected batched background migration for the given configuration to be marked as 'finished', but it is 'active':	{:job_class_name=>"CopyColumnUsingBackgroundMigrationJob", :table_name=>"ci_job_artifacts", :column_name=>"id", :job_arguments=>[["id", "job_id"], ["id_convert_to_bigint", "job_id_convert_to_bigint"]]}
gitlab    | 
gitlab    | Finalize it manualy by running
gitlab    | 
gitlab    | 	sudo gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,ci_job_artifacts,id,'[["id"\, "job_id"]\, ["id_convert_to_bigint"\, "job_id_convert_to_bigint"]]']
gitlab    | 
gitlab    | For more information, check the documentation
gitlab    | 
gitlab    | 	https://docs.gitlab.com/ee/user/admin_area/monitoring/background_migrations.html#database-migrations-failing-because-of-batched-background-migration-not-finished
gitlab    | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:1129:in `ensure_batched_background_migration_is_finished'
gitlab    | /opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20210706212710_finalize_ci_job_artifacts_bigint_conversion.rb:14:in `up'
gitlab    | /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:61:in `block (3 levels) in <top (required)>'
gitlab    | /opt/gitlab/embedded/bin/bundle:23:in `load'
gitlab    | /opt/gitlab/embedded/bin/bundle:23:in `<main>'
gitlab    | 
gitlab    | Caused by:
gitlab    | Expected batched background migration for the given configuration to be marked as 'finished', but it is 'active':	{:job_class_name=>"CopyColumnUsingBackgroundMigrationJob", :table_name=>"ci_job_artifacts", :column_name=>"id", :job_arguments=>[["id", "job_id"], ["id_convert_to_bigint", "job_id_convert_to_bigint"]]}
...
...
gitlab    |     System Info:
gitlab    |     ------------
gitlab    |     chef_version=15.14.0
gitlab    |     platform=ubuntu
gitlab    |     platform_version=20.04
gitlab    |     ruby=ruby 2.7.2p137 (2020-10-01 revision 5445e04352) [x86_64-linux]
gitlab    |     program_name=/opt/gitlab/embedded/bin/chef-client
gitlab    |     executable=/opt/gitlab/embedded/bin/chef-client
gitlab    |     
gitlab    | 
gitlab    | Running handlers:
gitlab    | Running handlers complete
gitlab    | Chef Infra Client failed. 11 resources updated in 28 seconds

I checked for background migrations before the upgrade, which returned 0:
gitlab-rails runner -e production ‘puts Gitlab::BackgroundMigration.remaining’

The docker container with gitlab keeps terminating and restarting, but i could successfully
run the command stated in the error message:
gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,ci_job_artifacts,id,’[[“id”, “job_id”], [“id_convert_to_bigint”, “job_id_convert_to_bigint”]]’]

But it kept restarting and logging the same error.
I tried again to run the migration, it said it was successful, but the same error reappeared.

I successfully downgraded to 4.1.0, because of the notes in the upgrading doc, which works. I did another attempt from 4.1.0 to 4.2.4, but it failed with the same error.
Then i downgraded again to 4.1.0, let the migration run
gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,ci_job_artifacts,id,’[[“id”, “job_id”], [“id_convert_to_bigint”, “job_id_convert_to_bigint”]]’]
upgraded to 4.1.5,
upgraded to 4.2.0,
upgraded to 4.2.4 successfully.

The Upgrade Path from the docs (latest 13.12.Z → latest 14.0.Z → latest 14.Y.Z) does not seem to be accurate.

asmith · September 22, 2021, 9:37pm

Hi @welano - did you follow our downgrade docs? There is a very specific order of commands you need to follow - additionally you will need a backup from 14.0.1 as part of the downgrade, so you can restore that. Downgrading via apt-get alone does not downgrade the database which is why the restore step is important.

Once you are on 14.0.1, you will need to upgrade to the latest 14.0.x release (which is 14.0.10 at the time of writing). Then, follow the link you shared Batched background migrations | GitLab and check the status of the batched background migration - once that is done it should be safe to update to 14.2 or later.

Here are some handy links for future reference:

Upgrade paths - the recommended upgrade path in your instance is latest 14.0.Z -> latest 14.Y.Z as indicated in that link
Version-specific upgrading instructions - any problems with specific versions are normally mentioned here (see 14.0.0 in particular)

asmith · September 22, 2021, 10:37pm

Hi @gork74 - this is definitely odd. Based on your post, I would agree you have followed our recommendations as written in our docs. Starting on 14.0.10 and checking that Background migrations and upgrades | GitLab was zero before proceeding to the latest 14.Y.Z should have worked.

Based on the rest of your post, it seems you were able to get upgraded to 14.2.Z successfully which is awesome!

The Upgrade Path from the docs (latest 13.12.Z → latest 14.0.Z → latest 14.Y.Z) does not seem to be accurate.

I did go and consult with my team and we are confident that the current upgrade path is correct latest 14.0.Z → latest 14.Y.Z. I’m not aware of any other users that needed to upgrade to 14.1.Z as an intermediary step but we will definitely keep an eye out. Thanks for posting your experiences though!

nondeterministic · September 23, 2021, 10:09am

FYI: I just ran into the same error on Ubuntu 18, coming from 13.8.8 => 13.12.10 => 14.0.10 => 14.3.0. The last step from 14.0.10 to 14.3.0 failed as described above. Running omnibus but not inside docker.

ministry-of-code · September 26, 2021, 3:12pm

We are facing the same problem. Upgrade from 13.12.10 to 14.0.0 worked fine, the following upgrade to the latest 14.3.x failed

Running handlers:
There was an error running gitlab-ctl reconfigure:

rails_migration[gitlab-rails] (gitlab::database_migrations line 51) had an error: Mixlib::ShellOut::ShellCommandFailed: bash[migrate gitlab-rails database] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/res
ources/rails_migration.rb line 16) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of "bash"  "/tmp/chef-script20210926-25225-1b1ec3z" ----
STDOUT: == 20210317210338 AddValidRunnerRegistrars: migrating =========================
-- add_column(:application_settings, :valid_runner_registrars, :string, {:array=>true, :default=>["project", "group"]})
   -> 0.0201s
== 20210317210338 AddValidRunnerRegistrars: migrated (0.0202s) ================

...


Expected batched background migration for the given configuration to be marked as 'finished', but it is 'active':	{:job_class_name=>"CopyColumnUsingBackgroundMigrationJob", :table_name=>"events", :column_name
=>"id", :job_arguments=>[["id"], ["id_convert_to_bigint"]]}

Finalize it manualy by running

	sudo gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,events,id,'[["id"]\, ["id_convert_to_bigint"]]']

For more information, check the documentation

Our installation was broken after that and we got a 500 SERVER ERROR after the login into Gitlab. I needed to downgrade the package to 14.0.0 and install the postgresql backup.

Would be great if someone could have a look at that, right now it seems not possible to upgrade.

iwalker · September 26, 2021, 5:21pm

I suggest doing this as your upgrade path:

Upgrade to latest 14.0.x release
Upgrade to latest 14.1.x release
Upgrade to latest 14.2.x release
Upgrade to 14.3.0.

Should work just make sure that background migrations finished before atarting the next upgrade.

fugit · September 27, 2021, 6:55pm

Just ran into this problem during our primary server upgrade(omnibus). The standby server did not have this problem doing the “same” migration path, likely do to there being almost no data on the standby server or possibly the small 14.0.Z difference.
primary server: 13.12.12 → 14.0.10 → 14.2.4 # failed with this error (upgrade was 20210925)
standby server: 13.12.12 → 14.0.11 → 14.2.4 # had no issues (14.0.11 was released today 20210927)

These checks were run before each update:

  gitlab-rake db:migrate:status | grep -v '   up '
  gitlab-rails runner -e production 'puts Gitlab::BackgroundMigration.remaining'
  gitlab-rake gitlab:check SANITIZE=true

Snowmanko · September 30, 2021, 2:08pm

Same problem here 13.12.12 → 14.0.11 → 14.3.0 failed. Should I downgrade?
I am also omnibus installation (CE) on Debian Buster.

ionutz · September 30, 2021, 7:04pm

Same as @Snowmanko
i tried from 13.12.12 to 14.0.11. then 14.1.* (worked ok). from 14.1.* to 14.2.* it breaks.

with all the precautions mentioned above, obviously.

ionutz · October 1, 2021, 7:40am

any updates or tips on how to get to latest version?

kinda stuck

sittingbool · October 1, 2021, 7:24pm

Hi, having the exact same issue as ionut.tanasa.commit and Snowmanko: breaks between 14.1.7-ee.0 and 14.2.5-ee.0

b1r63r · October 4, 2021, 3:12pm

I ran into this problem on centos. I am using the yum repository.
There seems to be a safe path going 14.1.7 → 14.2.1 → 14.2.5 → 14.3.1 (latest at the moment)
Going 14.1.7 → 14.2.5 (or 14.1.7 → 14.3.1 for that matter) fails.
Note that 14.2.0 has a problem with the rpm that makes it fail if you try using it for a yum update from earlier versions. That is the reason for using 14.2.1 in this chain
Whenever the updates failed I did a yum remove gitlab-ce and reinstalled the previous version. Then restored the database from the backup generated by the yum update.

asmith · October 6, 2021, 9:57pm

Following some further conversation I’ve had internally with our Database team, we have improved our documentation on how to recover from this error. The MR was merged 4 hours ago and is now live on our docs website.

Please see the following updated links:

Batched background migrations | GitLab

If you have any questions please let me know.

sittingbool · October 7, 2021, 4:04pm

I had the issue with docker on the ee - version. Is this fixed as well?

asmith · October 7, 2021, 4:37pm

Yes, the same instructions should work on all distributions of GitLab including Docker.

The problem occurs when batched migrations have not completed before upgrading from 14.0.5 - 14.0.Z or 14.1.Z to a higher version. As long as you have checked the status of the batched migrations and confirmed that they are all marked as finished before you upgrade to 14.2 and beyond, then the upgrade should be successful.

If the upgrade isn’t successful, the docs I mentioned previously (Batched background migrations | GitLab) will contain the next steps you can try.

sittingbool · October 7, 2021, 5:03pm

My point was: the update on docker runs by itself without me being able to interfere or try anything. Thats an automated process afaik.

ionutz · October 7, 2021, 8:52pm

i can confirm that this works. i was on 14.2.0 and went 14.2.1 → 14.2.5 → 14.3.1

ty very much sir @b1r63r

sittingbool · October 9, 2021, 12:46pm

Hi, I can also confirm that this path works for the gitlab/gitlab-ee docker release. I updated from 14.3.1 to 14.3.2 afterwards. That also worked fine.

outofcontrol · October 9, 2021, 1:52pm

Upgrading from source here:
13.12.12 → 14.0-stable
14-0-stable → 14-1-stable
14-1-stable → 14-2-stable → Background migration error suggests the Omnibus only fix:

sudo gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,ci_build_needs,id,'[["build_id"]\, ["build_id_convert_to_bigint"]]']

I’ve found the process to manually finish the background migrations for Omnibus installation (Batched background migrations | GitLab), but I don’t appear to find a work-around if upgrading from source.

On this page (Upgrading GitLab | GitLab) running:

sudo -u git -H bundle exec rails runner -e production 'puts Gitlab::BackgroundMigration.remaining'

shows 8 Background migrations remaining.

However, attempting to launch the rails console:

sudo -u git -H bundle exec rails RAILS_ENV=production

Results in:

Running the full GitLab test suite takes significant time to pass. We recommend using one of the following spec tasks: Learn more at https://docs.gitlab.com/ee/development/rake_tasks.html#run-tests.

Which is a page only for Omnibus.

Anyone know how I can finish the background migrations from source install of GitLab?