Upgraded to 14.7 but have pending migrations

How to Use GitLab Upgrade
1

We upgraded from 13 to 14.0.12 to 14.1.8 to 14.7 in one evening. We have a rather small deployment. After each upgrade we checked for background migrations. However between 14.1 and 14.7 I checked that remaining migrations had completed and returned 0 but not pending migrations list. Remaining was reporting 0. I was also confused by the batched background migrations viewable from the GUI which also had completed. Assuming that we were ok I ran the upgrade and the process completed successfully with no errors. However currently we are seeing jobs listed in the BackgroundMigrationJob.pending list. Every time I run this command I see two new hex numbers listed. TheBackgroundMigration.remaining reports 0 as does the Background Migrations in the GUI. Why would jobs be listed as pending but not remaining? Is there anything I can do to see a full list of these pending jobs? Do I need to roll back to 14.1 and verify the pending list finishes?

I was using this page for these commands. Note that remaining is returning 0 but not pending.

2

I had some still running after I upgraded to 14.7 from 14.6.3 but they went pretty quickly. The docs do mention that the pending/remaining jobs can take a while to process. How much time has passed since you did the upgrade?

I’ve never had jobs last as much as 24 hours before, but again could depend on server specifications and how much cpu/ram has been allocated to the server. Mine always generally finished within 10 minutes.

I did run the console remaining command yesterday or the day before when I upgraded, and it did report 3, but then they disappeared by the second time I ran the command. I would see what happens up to about 24 hours, and if still showing as pending then to start checking the log files I guess under /var/log/gitlab/gitlab-rails since this would most likely to be the place related to migrations and see if anything shows up in the log files.

3

So at this point were somewhere in the 36 hour neighborhood, but I’m still getting output from this command:

$ sudo gitlab-rails runner -e production ‘puts Gitlab::Database::BackgroundMigrationJob.pending’
#Gitlab::Database::BackgroundMigrationJob:0x00007fe48021c750
#Gitlab::Database::BackgroundMigrationJob:0x00007fe48013d320

And literally everytime the Hex numbers following BackgroundMigrationJob are different. But when I run this command I get 0

$ sudo gitlab-rails runner -e production ‘puts Gitlab::BackgroundMigration.remaining’
0

I checked logs for both gitlab-rails-db-migrate-date.log and migrations.log and niether has been updated today at all. I see no error messages in them from the prior day just a large list of completed migrations. Anything else I should be looking for?

4

@primordial Exact same picture here. Running 14.5.2 for a whole week now. Accidentally discovered there was a pending database migration job when I was about to upgrade to newer version. Any luck with this?

5

I have the same problem. I noticed this behavior in different versions of git lab (in my case 13.12.12 and 14.7). Those processes never go to zero and every time that runs the command a new HEX number is generented:

sudo gitlab-rails runner -e production ‘puts Gitlab::Database::BackgroundMigrationJob.pending’
#Gitlab::Database::BackgroundMigrationJob:0x00007f6c7c360570
#Gitlab::Database::BackgroundMigrationJob:0x00007f6c7da579e0
#Gitlab::Database::BackgroundMigrationJob:0x00007f6c7da57918

sudo gitlab-rails runner -e production ‘puts Gitlab::Database::BackgroundMigrationJob.pending’
#Gitlab::Database::BackgroundMigrationJob:0x00007fac8e130668
#Gitlab::Database::BackgroundMigrationJob:0x00007fac8e1966e8
#Gitlab::Database::BackgroundMigrationJob:0x00007fac8e196620

The others migrations checks go to 0 in minutes:

sudo gitlab-rails runner -e production ‘puts Gitlab::BackgroundMigration.remaining’
0
And all batched background migrations are with finished status.

I made a upgrade from version 13.12.12 to 14.7 ignoring this check after some hours waiting and did not notice any problems except the fact the database migration check never goes 0.

6

I was able to extract more information about the pending migration:

➜  ~ sudo gitlab-rails runner -e production 'puts Gitlab::Database::BackgroundMigrationJob.pending[0].pretty_inspect'
#<Gitlab::Database::BackgroundMigrationJob:0x00007f5d1c19d388
 id: 9,
 created_at: Sat, 23 Oct 2021 22:47:25.598178000 UTC +00:00,
 updated_at: Sat, 23 Oct 2021 22:47:25.598178000 UTC +00:00,
 status: "pending",
 class_name: "PopulateTopicsTotalProjectsCountCache",
 arguments: [1, 6]>
➜  ~
7

Same here. v14.0.12 to v14.8.0

I have Two Job

#<Gitlab::Database::BackgroundMigrationJob:0x00007f4e33a69c98
id: 16,
created_at: Wed, 23 Feb 2022 05:28:14.470755000 UTC +00:00,
updated_at: Wed, 23 Feb 2022 05:28:14.470755000 UTC +00:00,
status: “pending”,
class_name: “PopulateTopicsTotalProjectsCountCache”,
arguments: [1, 6]>
#<Gitlab::Database::BackgroundMigrationJob:0x00007fbfaa24e250
id: 19,
created_at: Wed, 23 Feb 2022 05:28:49.576125000 UTC +00:00,
updated_at: Wed, 23 Feb 2022 05:28:49.576125000 UTC +00:00,
status: “pending”,
class_name: “PopulateTopicsNonPrivateProjectsCount”,
arguments: [1, 6]>

I started from v13.6.7 to v13.8.8 to v13.12.15 to v14.0.12 to v14.8.0
All others went well.

8

I made all upgrade steps again 13.12.12 → 13.12.15 → 14.0.12 → 14.8.4

Until 14.0.12 update all job migrations finishes. Pending jobs occurs on 14.0.12 → 14.8.4 step.

gitlab-rails runner -e production ‘puts Gitlab::Database::BackgroundMigrationJob.pending[0].pretty_inspect’

#<Gitlab::Database::BackgroundMigrationJob:0x00007fa2ec1a6738
id: 26,
created_at: Tue, 22 Mar 2022 20:51:18.413634000 UTC +00:00,
updated_at: Tue, 22 Mar 2022 20:51:18.413634000 UTC +00:00,
status: “pending”,
class_name: “PopulateTopicsTotalProjectsCountCache”,
arguments: [1, 117]>

gitlab-rails runner -e production ‘puts Gitlab::Database::BackgroundMigrationJob.pending[1].pretty_inspect’

#<Gitlab::Database::BackgroundMigrationJob:0x00007f54e9fbad48
id: 37,
created_at: Tue, 22 Mar 2022 20:54:00.093371000 UTC +00:00,
updated_at: Tue, 22 Mar 2022 20:54:00.093371000 UTC +00:00,
status: “pending”,
class_name: “PopulateTopicsNonPrivateProjectsCount”,
arguments: [1, 117]>

1 Like
9

I have similar problem but in my case it is 14.1.8. The update path is 13.6.3 → 13.8.8 → 13.12.15 → 14.0.12 → 14.1.8.
I updated to 14.1.x because the page (Upgrading GitLab | GitLab) says

Upgrading to GitLab 14.5 (or later) may take a lot longer if you do not upgrade to at least 14.1 first.

All updated without error. And I checked the two commands, remaining and pending, returned 0 before each update.
Also on 14.0.12, I checked the web page showed ‘completed’ before update to 14.1.8.

Now, it is running 14.1.8 for weeks. The remaining command returned 0 but the pending command still returns 1.
The result of pretty_inspect is

#<Gitlab::Database::BackgroundMigrationJob:0x00007f82e86edfd0
 id: 5,
 created_at: Sun, 27 Feb 2022 06:57:18.594754000 UTC +00:00,
 updated_at: Sun, 27 Feb 2022 06:57:18.594754000 UTC +00:00,
 status: "pending",
 class_name: "BackfillDraftStatusOnMergeRequests",
 arguments: [167, 167]>

And there is no ‘down’ status in the following command. (Is it relevant?)
sudo gitlab-rake db:migrate:status

Is there anywhere to check, anything to try to fix this problem?

1 Like
10

Hello,
I have the same problem with PopulateTopicsTotalProjectsCountCache migration job while testing migration from 14.0.12 to 14.7.6. It is listed as pending whereas it seems to have successfully completed:

$ grep PopulateTopicsTotalProjectsCountCache /var/log/gitlab/sidekiq/current |grep ‘“job_status”:“done”’
{“severity”:“INFO”,“time”:“2022-03-28T06:49:59.005Z”,“retry”:3,“queue”:“background_migration”,“version”:0,“class”:“BackgroundMigrationWorker”,“args”:[“PopulateTopicsTotalProjectsCountCache”,“[1, 40]”],“jid”:“ccf21be2ff0d115f3ea9894c”,“created_at”:“2022-03-28T06:47:42.382Z”,“meta.caller_id”:“SchedulePopulateTopicsTotalProjectsCountCache”,“meta.feature_category”:“database”,“correlation_id”:“18040053f370d2bffac60d6aef8143b2”,“worker_data_consistency”:“always”,“scheduled_at”:“2022-03-28T06:49:42.382Z”,“idempotency_key”:“resque:gitlab:duplicate:background_migration:462b7d71e327c2df1e7f0ee6b13e36f2763e97726fac45574db2dbf4bdb970e1”,“enqueued_at”:“2022-03-28T06:49:58.878Z”,“job_size_bytes”:48,“pid”:18189,“message”:“BackgroundMigrationWorker JID-ccf21be2ff0d115f3ea9894c: done: 0.120601 sec”,“job_status”:“done”,“scheduling_latency_s”:0.00693,“enqueue_latency_s”:16.495442,“redis_calls”:3,“redis_duration_s”:0.003609,“redis_read_bytes”:15,“redis_write_bytes”:559,“redis_queues_calls”:1,“redis_queues_duration_s”:0.000183,“redis_queues_read_bytes”:10,“redis_queues_write_bytes”:338,“redis_shared_state_calls”:2,“redis_shared_state_duration_s”:0.003426,“redis_shared_state_read_bytes”:5,“redis_shared_state_write_bytes”:221,“db_count”:7,“db_write_count”:3,“db_cached_count”:0,“db_replica_count”:0,“db_primary_count”:7,“db_replica_cached_count”:0,“db_primary_cached_count”:0,“db_replica_wal_count”:0,“db_primary_wal_count”:0,“db_replica_wal_cached_count”:0,“db_primary_wal_cached_count”:0,“db_replica_duration_s”:0.0,“db_primary_duration_s”:0.029,“cpu_s”:0.013979,“mem_objects”:6781,“mem_bytes”:1349848,“mem_mallocs”:4706,“mem_total_bytes”:1621088,“duration_s”:0.120601,“completed_at”:“2022-03-28T06:49:59.005Z”,“load_balancing_strategy”:“primary”,“db_duration_s”:0.067683}

I have had the same kind of problem in the past during update to 13.8.8 with BackfillJiraTrackerDeploymentType2. This was solved with:

$ gitlab-rails console
Gitlab::Database::BackgroundMigrationJob.pending.where(class_name: “BackfillJiraTrackerDeploymentType2”).find_each do |job|
puts Gitlab::Database::BackgroundMigrationJob.mark_all_as_succeeded(“BackfillJiraTrackerDeploymentType2”, job.arguments)
end

See Upgrading GitLab | GitLab
Mathieu

11

Hello,
I have tried to reattempt the job as explained at Upgrading GitLab | GitLab but the perform method returns nil and the job is still pending:

irb(main):006:1* Gitlab::Database::BackgroundMigrationJob.pending.find_each do |job| 
irb(main):007:1*   puts "Running pending job '#{job.class_name}' with arguments #{job.arguments}" 
irb(main):008:1*   result = Gitlab::BackgroundMigration.perform(job.class_name, job.arguments) 
irb(main):009:1*   puts "Result: #{result}" 
irb(main):010:0> end 
Running pending job 'PopulateTopicsTotalProjectsCountCache' with arguments [1, 40] 
Result:  
=> nil

After viewing this thread Database Background Migration job Stuck, I have searched what this job is supposed to modify and it seems to run only one sql query in order to update counts in the topics table:
https://github.com/gitlabhq/gitlabhq/blob/master/lib/gitlab/background_migration/populate_topics_total_projects_count_cache.rb

In my case, the counts seem to be ok so I think I will go ahead and manually mark the job completed:

irb(main):011:0> Gitlab::Database::BackgroundMigrationJob.mark_all_as_succeeded(‘PopulateTopicsTotalProjectsCountCache’, [1, 40])
=> 1

Mathieu

12

Thanks Mathieu.
I have tried
Gitlab::BackgroundMigration.perform()
but because the result was nil, I forgot about it.

I searched ‘BackfillDraftStatusOnMergeRequests’ and what it is doing, then I concluded that is okay to mark as finished.
And run the following command.

irb(main):003:0> Gitlab::Database::BackgroundMigrationJob.mark_all_as_succeeded('BackfillDraftStatusOnMergeRequests', [167,167])
=> 1

https://www.rubydoc.info/github/gitlabhq/gitlabhq/Gitlab/BackgroundMigration/BackfillDraftStatusOnMergeRequests

13

Hello,
I have got an issue with the same background migration stuck at pending. Did you have any issues after setting it as completed?
Thanks

14

Hi!
I met as well.
I just used the next command for the killing stucked jobs

Gitlab::Database::BackgroundMigrationJob.pending.where(class_name: "ResetDuplicateCiRunnersTokenValuesOnProjects").find_each do |job|
  puts Gitlab::Database::BackgroundMigrationJob.mark_all_as_succeeded("ResetDuplicateCiRunnersTokenValuesOnProjects", job.arguments)
end

it was from there:

Just curious will it be a problem in the future