Background migrations stuck after running gitlab:check

How to Use GitLab Upgrade
1

I upgraded a GitLab omnibus install to v18.5.0 and before the background migrations were done I ran sudo gitlab-rake gitlab:check as part of my checklist to wrap up the upgrade, and I noticed that after that the background migrations stopped running after that.

There are currently 12 migrations sitting in the queue, but I haven’t been able to figure out how to get them to resume running even after reading Troubleshooting background migrations | GitLab Docs

Tried the following

  • Restarting sudo gitlab-ctl restart
  • Running sudo gitlab-rake db:migrate
  • In the UI, pausing and resuming one of the jobs

In the logs, I see this repeatedly, but considering a restart didn’t fix it, it’s not clear what to do next.

==> /var/log/gitlab/sidekiq/current <==
[2025-10-17 15:31:53] INFO  WEBrick 1.9.1
[2025-10-17 15:31:53] INFO  ruby 3.2.8 (2025-03-26) [x86_64-linux]
Address already in use - bind(2) for 127.0.0.1:8092
/opt/gitlab/embedded/lib/ruby/3.2.0/socket.rb:205:in `bind'
/opt/gitlab/embedded/lib/ruby/3.2.0/socket.rb:205:in `listen'
/opt/gitlab/embedded/lib/ruby/3.2.0/socket.rb:768:in `block in tcp_server_sockets'
/opt/gitlab/embedded/lib/ruby/3.2.0/socket.rb:231:in `each'
/opt/gitlab/embedded/lib/ruby/3.2.0/socket.rb:231:in `foreach'
/opt/gitlab/embedded/lib/ruby/3.2.0/socket.rb:766:in `tcp_server_sockets'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/webrick-1.9.1/lib/webrick/utils.rb:60:in `create_listeners'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/webrick-1.9.1/lib/webrick/server.rb:130:in `listen'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/webrick-1.9.1/lib/webrick/server.rb:111:in `initialize'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/webrick-1.9.1/lib/webrick/httpserver.rb:47:in `initialize'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/health_checks/server.rb:18:in `new'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/health_checks/server.rb:18:in `start_working'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/daemon.rb:55:in `block in start'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/daemon.rb:52:in `synchronize'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/daemon.rb:52:in `start'
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/sidekiq.rb:93:in `block (2 levels) in <top (required)>'
/opt/gitlab/embedded/service/gitlab-rails/vendor/gems/sidekiq/lib/sidekiq/component.rb:60:in `block in fire_event'
/opt/gitlab/embedded/service/gitlab-rails/vendor/gems/sidekiq/lib/sidekiq/component.rb:59:in `each'
/opt/gitlab/embedded/service/gitlab-rails/vendor/gems/sidekiq/lib/sidekiq/component.rb:59:in `fire_event'
/opt/gitlab/embedded/service/gitlab-rails/vendor/gems/sidekiq/lib/sidekiq/cli.rb:108:in `run'
/opt/gitlab/embedded/service/gitlab-rails/vendor/gems/sidekiq/bin/sidekiq:31:in `<top (required)>'
/opt/gitlab/embedded/lib/ruby/site_ruby/3.2.0/rubygems.rb:319:in `load'
/opt/gitlab/embedded/lib/ruby/site_ruby/3.2.0/rubygems.rb:319:in `activate_and_load_bin_path'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/bin/sidekiq:25:in `<top (required)>'
/opt/gitlab/embedded/lib/ruby/site_ruby/3.2.0/bundler/cli/exec.rb:59:in `load'
/opt/gitlab/embedded/lib/ruby/site_ruby/3.2.0/bundler/cli/exec.rb:59:in `kernel_load'
/opt/gitlab/embedded/lib/ruby/site_ruby/3.2.0/bundler/cli/exec.rb:23:in `run'
/opt/gitlab/embedded/lib/ruby/site_ruby/3.2.0/bundler/cli.rb:451:in `exec'
/opt/gitlab/embedded/lib/ruby/site_ruby/3.2.0/bundler/vendor/thor/lib/thor/command.rb:28:in `run'
/opt/gitlab/embedded/lib/ruby/site_ruby/3.2.0/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
/opt/gitlab/embedded/lib/ruby/site_ruby/3.2.0/bundler/vendor/thor/lib/thor.rb:538:in `dispatch'
/opt/gitlab/embedded/lib/ruby/site_ruby/3.2.0/bundler/cli.rb:35:in `dispatch'
/opt/gitlab/embedded/lib/ruby/site_ruby/3.2.0/bundler/vendor/thor/lib/thor/base.rb:584:in `start'
/opt/gitlab/embedded/lib/ruby/site_ruby/3.2.0/bundler/cli.rb:29:in `start'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/bundler-2.7.1/exe/bundle:28:in `block in <top (required)>'
/opt/gitlab/embedded/lib/ruby/site_ruby/3.2.0/bundler/friendly_errors.rb:118:in `with_friendly_errors'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/bundler-2.7.1/exe/bundle:20:in `<top (required)>'
/opt/gitlab/embedded/lib/ruby/site_ruby/3.2.0/rubygems.rb:319:in `load'
/opt/gitlab/embedded/lib/ruby/site_ruby/3.2.0/rubygems.rb:319:in `activate_and_load_bin_path'
/opt/gitlab/embedded/bin/bundle:25:in `<main>'
{"severity":"INFO","time":"2025-10-17T15:31:53.598Z","message":"stopped","memwd_reason":"background task stopped","memwd_handler_class":"Gitlab::Memory::Watchdog::Handlers::SidekiqHandler","memwd_sleep_time_s":3,"pid":1827700,"worker_id":"sidekiq_0","memwd_rss_bytes":1176383488,"retry":0}
2

I’ve tried running sudo gitlab-rake gitlab:sidekiq:migrate_jobs:retry gitlab:sidekiq:migrate_jobs:schedule gitlab:sidekiq:migrate_jobs:queued which had no impact.

So ended up following the steps at Check migrations before upgrade | GitLab Docs to do a database query for the stuck migrations, and then ran sudo gitlab-rake gitlab:background_migrations:finalize[<job_class_name>,<table_name>,<column_name>,‘<job_arguments>’] for each of them.

It was a bit tedious but got the job done.

1 Like