Upgrade Problems Debian 14.2.4

Following some further conversation I’ve had internally with our Database team, we have improved our documentation on how to recover from this error. The MR was merged 4 hours ago and is now live on our docs website.

Please see the following updated links:

If you have any questions please let me know.

I had the issue with docker on the ee - version. Is this fixed as well?

Yes, the same instructions should work on all distributions of GitLab including Docker.

The problem occurs when batched migrations have not completed before upgrading from 14.0.5 - 14.0.Z or 14.1.Z to a higher version. As long as you have checked the status of the batched migrations and confirmed that they are all marked as finished before you upgrade to 14.2 and beyond, then the upgrade should be successful.

If the upgrade isn’t successful, the docs I mentioned previously (Batched background migrations | GitLab) will contain the next steps you can try.

1 Like

My point was: the update on docker runs by itself without me being able to interfere or try anything. Thats an automated process afaik.

i can confirm that this works. i was on 14.2.0 and went 14.2.1 → 14.2.5 → 14.3.1

ty very much sir @b1r63r

Hi, I can also confirm that this path works for the gitlab/gitlab-ee docker release. I updated from 14.3.1 to 14.3.2 afterwards. That also worked fine.

Upgrading from source here:
13.12.12 → 14.0-stable
14-0-stable → 14-1-stable
14-1-stable → 14-2-stable → Background migration error suggests the Omnibus only fix:

sudo gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,ci_build_needs,id,'[["build_id"]\, ["build_id_convert_to_bigint"]]']

I’ve found the process to manually finish the background migrations for Omnibus installation (Batched background migrations | GitLab), but I don’t appear to find a work-around if upgrading from source.

On this page (Upgrading GitLab | GitLab) running:

sudo -u git -H bundle exec rails runner -e production 'puts Gitlab::BackgroundMigration.remaining'

shows 8 Background migrations remaining.

However, attempting to launch the rails console:

sudo -u git -H bundle exec rails RAILS_ENV=production

Results in:

Running the full GitLab test suite takes significant time to pass. We recommend using one of the following spec tasks: Learn more at https://docs.gitlab.com/ee/development/rake_tasks.html#run-tests.

Which is a page only for Omnibus.

Anyone know how I can finish the background migrations from source install of GitLab?

Using the correct command rails console command helped:

sudo -u git -H bundle exec rails console -e production

Then executing:

scheduled_queue = Sidekiq::ScheduledSet.new
pending_job_classes = scheduled_queue.select { |job| job["class"] == "BackgroundMigrationWorker" }.map { |job| job["args"].first }.uniq
pending_job_classes.each { |job_class| Gitlab::BackgroundMigration.steal(job_class) }

Appears to successfully finalize the background migrations as running

sudo -u git -H bundle exec rails runner -e production 'puts Gitlab::BackgroundMigration.remaining'

results in 0.

However, I still get the same error when attempting to finish the migration. Seems I still need to run the equivalent of this in the rails console:

gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,ci_build_needs,id,'[["build_id"]\, ["build_id_convert_to_bigint"]]']

Not sure how to do this. Anyone have a suggestion? Beginning to think I need to downgrade to 14.1 :frowning:

Okay, seems like my issue is impatience. After 6 hours the background migrations are still running as seen in the GitLab Admin Monitoring area. Just not showing on the command line.

Hi,

I’ve just ran into the same problem as described above. (I’m on CentOS 7, using the yum repo). I was coming from 13.x, so first try the yum error told me to upgrade to 14.0.x (latest), I did, then I just ran the next without specifying the version, so it jumped to 14.5.x, but had this error, the whole instance didn’t start. After like 2hrs of tries, while it kept asking for the same thing all the time:

gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,events,id,'[["id"]\, ["id_convert_to_bigint"]]']

I could finally could run the below:

gitlab-rake db:migrate
gitlab-ctl reconfigure

I’m not too sure what fixed it (maybe gitlab-rake gitlab:check SANITIZE=true, but I ran a lot of commands, had 0 in the queue), but it looks ok now. Can you please add a check to the RPMs? So it won’t let us upgrade from 14.0.x → 14.5, but should prompt for the correct journey, 14.1.x…

Thanks!

Any update on this please? Should I report in somewhere as a bug?

You didn’t follow the upgrade path, you jumped. You will have to go back to 14.0.x, restore the backup and then start the upgrade properly following the upgrade path outlined in the documentation as well as earlier in this post: Upgrade Problems Debian 14.2.4 - #7 by iwalker

此问题我已经解决,并且我已经成功升级到14.5.2
我的升级路线为
10.7->10.8.7 → 11.11.8 → 12.0.12 → 12.1.17 → 12.10.14 → 13.0.14 → 13.1.11 → 13.8.8
→ 13.9.2-> 13.12.12 → 14.0.11 → 14.1.6->14.1.7->14.2.1->14.2.5->14.2.6->14.2.7-> 14.3.4->14.4.3->14.5.2

14.2.X 升级到14.3.X 的时候需要更新PG 的数据,但是需要升级到14.2.7 或者14.2.6 进入容器
此时应该运行14.2.7 docker exec -ti gitlab
gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,push_event_payloads,event_id,’[[“event_id”], [“event_id_convert_to_bigint”]]’]
gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,ci_stages,id,’[[“id”], [“id_convert_to_bigint”]]’]

gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,events,id,’[[“id”], [“id_convert_to_bigint”]]’]

时间会比较慢,但是需要耐心等待,在14.2.7 上执行成功之后,在升级14.3 或者14.5 版本就可以了,希望帮助到大家

Hello, I ran into the same problem. my migration path was:
12.10.14 → 13.0.14 → 13.1.11 → 13.8.8 → 13.12.15 → 14.0.11 → 14.1.8 → 14.2.6
like recommended on Upgrading GitLab | GitLab

During the update from 14.1.8 to 14.2.6 i got the Problem
I am running gitlab in docker.

Is there any update for this issue?

I solved the problem for me. I started gitlab with version 14.1.8 and used the following commands from Batched background migrations | GitLab
$ gitlab-rake db:migrate
$ gitlab-ctl reconfigure
inside my docker container.
After that i changed the version to 14.2.6, it took about a minute and everything seems fine

1 Like

I’m facing the same issue. I created a forum post a while back describing my situation but I have yet to receive any help.

I’m having the same issue with an Omnibus installation. I’ve attempted the smallest upgrade from Docker image gitlab/gitlab-ce:14.1.8-ce.0 to gitlab/gitlab-ce:14.2.0-ce.0, I’ve also tried a few other releases in the 14.2.x range, however the result is the same.

The Docker container restarts every 30 seconds or so (before the database migrations can complete). The logs indicate the following:

n error has occurred, all later migrations canceled:

Expected batched background migration for the given configuration to be marked as ‘finished’, but it is ‘finalizing’: {:job_class_name=>“CopyColumnUsingBackgroundMigrationJob”, :table_name=>“ci_job_artifacts”, :column_name=>“id”, :job_arguments=>[[“id”, “job_id”], [“id_convert_to_bigint”, “job_id_convert_to_bigint”]]}

Finalize it manualy by running

sudo gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,ci_job_artifacts,id,’[[“id”, “job_id”], [“id_convert_to_bigint”, “job_id_convert_to_bigint”]]’]

For more information, check the documentation

Batched background migrations | GitLab

However 30 seconds is insufficient to execute the command.

Any advice on how to get past this?

I was stucked with14.1.8 to 14.2.x, reading above, I came with the commands which finally works for me.

Run with 14.1.8, inside the docker and run the commands before upgrade to 14.2.1

gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,events,id,'[["id"]\, ["id_convert_to_bigint"]]']
gitlab-rails runner -e production 'puts Gitlab::BackgroundMigration.remaining'
gitlab-rake gitlab:check SANITIZE=true
gitlab-rake db:migrate
gitlab-ctl reconfigure

My upgrade path

14.1.8 (run with above commands) → 14.2.1 → 14.2.5 → 14.3.1 → 14.3.6 → 14.6.2 →
14.9.5 → 14.10.5 → 15.0.2 → 15.1.0 (latest major version at the moment)

Thank you @weisunding , I confirm that the step from 14.1.8 and the upgrade path works. You are the best