Pending migration for over 24 hours

Despite the background migrations showing zero either on the console or web interface, when checking pending count for migrations, one has been showing for over 24 hours.

When running the following:

gitlab-rails runner -e production 'puts Gitlab::Database::BackgroundMigrationJob.pending[0].pretty_inspect'

#<Gitlab::Database::BackgroundMigrationJob:0x00007fb32d714228
 id: 12,
 created_at: Tue, 22 Mar 2022 21:32:38.517548000 UTC +00:00,
 updated_at: Tue, 22 Mar 2022 21:32:38.517548000 UTC +00:00,
 status: "pending",
 class_name: "ResetDuplicateCiRunnersTokenValuesOnProjects",
 arguments: [2, 9]>

It shows this detail. I’ve tried running:

gitlab-rails db:migrate

but nothing changes, also all others show as being up when checking status:

   up     20220308115502  Schedule reset duplicate ci runners token values on projects

How can this pending one be forced or removed? I have tried what is in the docs: Upgrading GitLab | GitLab

gitlab-rails c
--------------------------------------------------------------------------------
 Ruby:         ruby 2.7.5p203 (2021-11-24 revision f69aeb8314) [x86_64-linux]
 GitLab:       14.9.1 (5b713e1e9a3) FOSS
 GitLab Shell: 13.24.0
 PostgreSQL:   12.7
------------------------------------------------------------[ booted in 40.01s ]

Loading production environment (Rails 6.1.4.6)
irb(main):001:0> 
irb(main):002:0> scheduled_queue = Sidekiq::ScheduledSet.new
=> #<Sidekiq::ScheduledSet:0x00007f53f240f810 @name="schedule", @_size=2>
irb(main):003:0> pending_job_classes = scheduled_queue.select { |job| job["class"] == "BackgroundMigrationWorker" }.map { |job| job
["args"].first }.uniq
=> []
irb(main):004:0> pending_job_classes.each { |job_class| Gitlab::BackgroundMigration.steal(job_class) }
=> []
irb(main):005:1* Gitlab::Database::BackgroundMigrationJob.pending.find_each do |job|
irb(main):006:1*   puts "Running pending job '#{job.class_name}' with arguments #{job.arguments}"
irb(main):007:1*   result = Gitlab::BackgroundMigration.perform(job.class_name, job.arguments)
irb(main):008:1*   puts "Result: #{result}"
irb(main):009:0> end
Running pending job 'ResetDuplicateCiRunnersTokenValuesOnProjects' with arguments [2, 9]
Result: 0
=> nil

and doesn’t help, output from pending:

gitlab-rails runner -e production 'puts Gitlab::Database::BackgroundMigrationJob.pending.count
1

It seems like it has ran due to the migration being up, but it still shows in the pending count.

From checking the psql table:

gitlabhq_production=# select * from background_migration_jobs;
 id |          created_at           |          updated_at           | status |                                class_name           
                     |                            arguments                            
----+-------------------------------+-------------------------------+--------+-----------------------------------------------------
---------------------+-----------------------------------------------------------------
  1 | 2020-09-11 07:53:56.948512+00 | 2020-09-11 07:56:02.755423+00 |      1 | Gitlab::Database::PartitioningMigrationHelpers::Back
fillPartitionedTable | [1, 1706, "audit_events", "audit_events_part_5fc467ac26", "id"]
  4 | 2021-05-24 12:31:10.668441+00 | 2021-05-24 12:33:12.410308+00 |      1 | BackfillNamespaceTraversalIdsRoots                  
                     | [1, 31, 100]
  5 | 2021-07-22 20:29:05.751912+00 | 2021-07-22 20:31:11.180535+00 |      1 | MigrateMergeRequestDiffCommitUsers                  
                     | [1, 40001]
 10 | 2022-01-24 11:42:26.819665+00 | 2022-01-24 11:44:29.253568+00 |      1 | BackfillIncidentIssueEscalationStatuses             
                     | [46, 1786]
 12 | 2022-03-22 21:32:38.517548+00 | 2022-03-22 21:32:38.517548+00 |      0 | ResetDuplicateCiRunnersTokenValuesOnProjects        
                     | [2, 9]
 11 | 2022-03-22 21:32:38.315361+00 | 2022-03-22 21:34:47.424816+00 |      1 | ResetDuplicateCiRunnersTokenEncryptedValuesOnProject
s                    | [2, 18]
(6 rows)

I guess potentially I could manually change the status from 0 to 1. But would prefer a cleaner method if possible.

If anyone has any ideas, would be great, but am still curious how it can be pending, and yet show as completed (up).

So, in the end I just did this:

gitlabhq_production=# update background_migration_jobs set status=1 where id=12;

considering that it was already showing as up would mean that it had finished successfully and just not changed the status in the database for some reason. At least that is what I’m assuming. I guess I’ll find out in the future or not when upgrading if any future problems occur relating to this or not.

I will mark this as a potential solution for now.

I had the same issue when upgrading from omnibus version 14.10.5-ee to 15.0.5-ee.

The content in this topic was very helpful. I ran the same command in my environment and contacted tech support about the situation.
Support gave me the following solution.

This is a known bug with ResetDuplicateCiRunnersTokenValuesOnProjects. You can try the following steps to resolve the issue.

Gitlab::Database::BackgroundMigrationJob.pending.where(class_name: "ResetDuplicateCiRunnersTokenValuesOnProjects").find_each do |job|
  puts Gitlab::Database::BackgroundMigrationJob.mark_all_as_succeeded("ResetDuplicateCiRunnersTokenValuesOnProjects", job.arguments)

After doing the steps, the issue was resolved.

gitlab@gitlab-desktop:~$ sudo gitlab-rails runner -e production 'puts Gitlab::Database::BackgroundMigrationJob.pending.count'
[sudo] password for gitlab :
1
gitlab@gitlab-desktop:~$ sudo gitlab-rails c
--------------------------------------------------------------------------------
 Ruby:         ruby 2.7.5p203 (2021-11-24 revision f69aeb8314) [x86_64-linux]
 GitLab:       14.10.5-ee (af6735ad95f) EE
 GitLab Shell: 13.25.1
 PostgreSQL:   12.7
------------------------------------------------------------[ booted in 57.10s ]
Loading production environment (Rails 6.1.4.7)
irb(main):004:1* Gitlab::Database::BackgroundMigrationJob.pending.where(class_name: "ResetDuplicateCiRunnersTokenValuesOnProjects").find_each do |job|
irb(main):005:1*   puts Gitlab::Database::BackgroundMigrationJob.mark_all_as_succeeded("ResetDuplicateCiRunnersTokenValuesOnProjects", job.arguments)
irb(main):006:0> end
1
=> nil
irb(main):007:0> exit
gitlab@gitlab-desktop:~$ sudo gitlab-rails runner -e production 'puts Gitlab::Database::BackgroundMigrationJob.pending.count'
0
gitlab@gitlab-desktop:~$ sudo gitlab-rails runner -e production 'puts Gitlab::BackgroundMigration.remaining'
0
gitlab@gitlab-desktop:~$

I hope this information helps someone.