Upgrade from 15.0.4 to 15.1.0 fails docker

Hello I am running a gitlab-ce serveer using docker I just updated from 14.10.0 to 14.10.5 then to 15.0.4 which worked fine but I can’t upgrade to 15.1 I tried to upgrade to the last build 15.1.2 it didn’t work then I tried to 15.1.0 and I get the same error

Running handlers:
gitlab-ce | [2022-07-18T11:48:29+00:00] ERROR: Running exception handlers
gitlab-ce | There was an error running gitlab-ctl reconfigure:
gitlab-ce |
gitlab-ce | rails_migration[gitlab-rails] (gitlab::database_migrations line 51) had an error: Mixlib::ShellOut::ShellCommandFailed: bash[migrate gitlab-rails database] (gitlab::database_migrations line 18) had an error: Mixlib::ShellOut::ShellCommandFailed: Command execution failed. STDOUT/STDERR suppressed for sensitive resource
gitlab-ce |
gitlab-ce | Running handlers complete
gitlab-ce | [2022-07-18T11:48:29+00:00] ERROR: Exception handlers complete
gitlab-ce | Infra Phase failed. 163 resources updated in 01 minutes 20 seconds
gitlab-ce | [2022-07-18T11:48:29+00:00] FATAL: Stacktrace dumped to /opt/gitlab/embedded/cookbooks/cache/chef-stacktrace.out
gitlab-ce | [2022-07-18T11:48:29+00:00] FATAL: ---------------------------------------------------------------------------------------
gitlab-ce | [2022-07-18T11:48:29+00:00] FATAL: PLEASE PROVIDE THE CONTENTS OF THE stacktrace.out FILE (above) IF YOU FILE A BUG REPORT
gitlab-ce | [2022-07-18T11:48:29+00:00] FATAL: ---------------------------------------------------------------------------------------

root@gitlab:/# cat /opt/gitlab/embedded/cookbooks/cache/chef-stacktrace.out
Generated at 2022-07-18 11:50:59 +0000
Mixlib::ShellOut::ShellCommandFailed: rails_migration[gitlab-rails] (gitlab::database_migrations line 51) had an error: Mixlib::ShellOut::ShellCommandFailed: bash[migrate gitlab-rails database] (gitlab::database_migrations line 18) had an error: Mixlib::ShellOut::ShellCommandFailed: Command execution failed. STDOUT/STDERR suppressed for sensitive resource
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/provider/execute.rb:60:in rescue in block (2 levels) in <class:Execute>' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/provider/execute.rb:51:in block (2 levels) in class:Execute’
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/mixin/why_run.rb:51:in add_action' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/provider.rb:293:in converge_by’
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/provider/execute.rb:50:in block in <class:Execute>' (eval):2:in block in action_run’
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/provider.rb:304:in instance_eval' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/provider.rb:304:in compile_and_converge_action’
(eval):2:in action_run' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/provider.rb:245:in run_action’
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/resource.rb:601:in block in run_action' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/resource.rb:628:in with_umask’
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/resource.rb:600:in run_action' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/runner.rb:74:in run_action’
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/runner.rb:108:in block in run_all_actions' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/runner.rb:108:in each’
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/runner.rb:108:in run_all_actions' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/resource_collection.rb:64:in insert’
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/dsl/declare_resource.rb:267:in declare_resource' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/dsl/resources.rb:36:in bash’
/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb:18:in block in class_from_file' (eval):2:in block in action_run’
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/provider.rb:304:in instance_eval' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/provider.rb:304:in compile_and_converge_action’
(eval):2:in action_run' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/provider.rb:245:in run_action’
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/resource.rb:601:in block in run_action' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/resource.rb:628:in with_umask’
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/resource.rb:600:in run_action' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/runner.rb:74:in run_action’
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/runner.rb:108:in block in run_all_actions' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/runner.rb:108:in each’
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/runner.rb:108:in run_all_actions' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/runner.rb:132:in block in converge’
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/resource_collection/resource_list.rb:96:in block in execute_each_resource' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/resource_collection/stepable_iterator.rb:114:in call_iterator_block’
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/resource_collection/stepable_iterator.rb:85:in step' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/resource_collection/stepable_iterator.rb:103:in iterate’
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/resource_collection/stepable_iterator.rb:54:in each_with_index' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/resource_collection/resource_list.rb:94:in execute_each_resource’
/opt/gitlab/embedded/lib/ruby/2.7.0/forwardable.rb:235:in execute_each_resource' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/runner.rb:130:in converge’
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/client.rb:686:in block in converge' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/client.rb:681:in catch’
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/client.rb:681:in converge' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/client.rb:705:in converge_and_save’
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/client.rb:285:in run' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/application.rb:305:in run_with_graceful_exit_option’
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/application.rb:281:in block in run_chef_client' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/local_mode.rb:42:in with_server_connectivity’
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/application.rb:264:in run_chef_client' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/application/base.rb:352:in run_application’
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-17.10.0/lib/chef/application.rb:67:in run' /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-bin-17.10.0/bin/cinc-client:25:in <top (required)>’
/opt/gitlab/embedded/bin/cinc-client:23:in load' /opt/gitlab/embedded/bin/cinc-client:23:in 'root@gitlab:/#

Any idea ?
Thank you

I got exactly the same problem here - tried different versions with no different result

Okay, here is a hotfix: while the container was starting I deleted the /opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20220601040233_finalize_backfill_issue_search_data.rb patch file, which caused the error and the update went trough. Now I need to figure out what the post migration script was trying to do.

You can skip the script by executing: gitlab-rake gitlab:db:mark_migration_complete[20220601040233] in the running docker container

At last check your GUI: https://YOURURL/admin/background_migrations there you may find the background task BackfillIssueSearchData which failed. For me just restarting it got the job done after all

Thank you
I will try this and let you informed.

Thanks MrSchizo, I was able to upgrade to 15.1.4 with your work around and then update to 15.2.2 went smoothly and same here I just restarted failed migration jobs and it succeeded