Fail when I upgrade gitlab from 16.8.0 to 16.9.1

:hugs: Please help fill in this template with all the details to help others help you more efficiently. Use formatting blocks for code, config, logs and ensure to remove sensitive data.

Problem to solve

I upgrade gitlab from 16.8.0 to 16.9.1, when I execute the command “gitlab-ctl reconfigure”
,cinc-stacktrace.out is shown below:

Generated at 2024-02-27 15:56:59 +0800
Mixlib::ShellOut::ShellCommandFailed: rails_migration[gitlab-rails] (gitlab::database_migrations line 51) had an error: Mixlib::ShellOut::ShellCommandFailed: bash_hide_env[migrate gitlab-rails database] (gitlab::database_migrations line 20) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received ‘1’
---- Begin output of “bash” ----
STDOUT: rake aborted!
Feature::InvalidFeatureFlagError: Feature flag ‘ci_job_artifacts_backlog_large_loop_limit’ is already defined in ‘/opt/gitlab/embedded/service/gitlab-rails/config/feature_flags/development/ci_job_artifacts_backlog_large_loop_limit.yml’
/opt/gitlab/embedded/service/gitlab-rails/lib/feature/definition.rb:194:in block in load_all_from_path!' /opt/gitlab/embedded/service/gitlab-rails/lib/feature/definition.rb:190:in each’
/opt/gitlab/embedded/service/gitlab-rails/lib/feature/definition.rb:190:in load_all_from_path!' /opt/gitlab/embedded/service/gitlab-rails/lib/feature/definition.rb:175:in block in load_all!’
/opt/gitlab/embedded/service/gitlab-rails/lib/feature/definition.rb:174:in each' /opt/gitlab/embedded/service/gitlab-rails/lib/feature/definition.rb:174:in each_with_object’
/opt/gitlab/embedded/service/gitlab-rails/lib/feature/definition.rb:174:in load_all!' /opt/gitlab/embedded/service/gitlab-rails/lib/feature/definition.rb:119:in reload!’
/opt/gitlab/embedded/service/gitlab-rails/lib/feature.rb:254:in register_definitions' /opt/gitlab/embedded/service/gitlab-rails/config/initializers/0_inject_feature_flags.rb:7:in <top (required)>’
/opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:7:in <top (required)>' <internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:37:in require’
internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb:37:in require' /opt/gitlab/embedded/bin/bundle:25:in load’
/opt/gitlab/embedded/bin/bundle:25:in <main>' Tasks: TOP => gitlab:db:configure => environment (See full trace by running task with --trace) STDERR: ---- End output of "bash" ---- Ran "bash" returned 1 /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/mixlib-shellout-3.2.7/lib/mixlib/shellout.rb:300:in invalid!’
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/mixlib-shellout-3.2.7/lib/mixlib/shellout.rb:287:in error!' /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/mixlib-shellout-3.2.7/lib/mixlib/shellout/helper.rb:130:in shell_out_compacted!’
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/mixlib-shellout-3.2.7/lib/mixlib/shellout/helper.rb:54:in shell_out!' /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/provider/execute.rb:52:in block (2 levels) in class:Execute
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/mixin/why_run.rb:51:in add_action' /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/provider.rb:293:in converge_by’
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/provider/execute.rb:50:in block in <class:Execute>' (eval):2:in block in action_run’
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/provider.rb:304:in instance_eval' /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/provider.rb:304:in compile_and_converge_action’
(eval):2:in action_run' /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/provider.rb:245:in run_action’
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/resource.rb:601:in block in run_action' /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/resource.rb:628:in with_umask’
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/resource.rb:600:in run_action' /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/runner.rb:74:in run_action’
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/runner.rb:108:in block in run_all_actions' /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/runner.rb:108:in each’
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/runner.rb:108:in run_all_actions' /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/resource_collection.rb:64:in insert’
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/dsl/declare_resource.rb:267:in declare_resource' /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/dsl/resources.rb:36:in bash_hide_env’
/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb:20:in block in class_from_file' (eval):2:in block in action_run’
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/provider.rb:304:in instance_eval' /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/provider.rb:304:in compile_and_converge_action’
(eval):2:in action_run' /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/provider.rb:245:in run_action’
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/resource.rb:601:in block in run_action' /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/resource.rb:628:in with_umask’
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/resource.rb:600:in run_action' /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/runner.rb:74:in run_action’
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/runner.rb:108:in block in run_all_actions' /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/runner.rb:108:in each’
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/runner.rb:108:in run_all_actions' /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/runner.rb:132:in block in converge’
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/resource_collection/resource_list.rb:96:in block in execute_each_resource' /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/resource_collection/stepable_iterator.rb:114:in call_iterator_block’
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/resource_collection/stepable_iterator.rb:85:in step' /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/resource_collection/stepable_iterator.rb:103:in iterate’
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/resource_collection/stepable_iterator.rb:54:in each_with_index' /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/resource_collection/resource_list.rb:94:in execute_each_resource’
/opt/gitlab/embedded/lib/ruby/3.1.0/forwardable.rb:238:in execute_each_resource' /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/runner.rb:130:in converge’
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/client.rb:875:in block in converge' /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/client.rb:870:in catch’
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/client.rb:870:in converge' /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/client.rb:894:in converge_and_save’
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/client.rb:298:in run' /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/application.rb:305:in run_with_graceful_exit_option’
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/application.rb:281:in block in run_chef_client' /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/local_mode.rb:42:in with_server_connectivity’
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/application.rb:264:in run_chef_client' /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/application/base.rb:354:in run_application’
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/application.rb:67:in run' /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-bin-18.3.0/bin/cinc-client:25:in <top (required)>’
/opt/gitlab/embedded/bin/cinc-client:25:in load' /opt/gitlab/embedded/bin/cinc-client:25:in

Caused by Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received ‘1’
---- Begin output of “bash” ----
STDOUT: rake aborted!
Feature::InvalidFeatureFlagError: Feature flag ‘ci_job_artifacts_backlog_large_loop_limit’ is already defined in ‘/opt/gitlab/embedded/service/gitlab-rails/config/feature_flags/development/ci_job_artifacts_backlog_large_loop_limit.yml’
/opt/gitlab/embedded/service/gitlab-rails/lib/feature/definition.rb:194:in block in load_all_from_path!' /opt/gitlab/embedded/service/gitlab-rails/lib/feature/definition.rb:190:in each’
/opt/gitlab/embedded/service/gitlab-rails/lib/feature/definition.rb:190:in load_all_from_path!' /opt/gitlab/embedded/service/gitlab-rails/lib/feature/definition.rb:175:in block in load_all!’
/opt/gitlab/embedded/service/gitlab-rails/lib/feature/definition.rb:174:in each' /opt/gitlab/embedded/service/gitlab-rails/lib/feature/definition.rb:174:in each_with_object’
/opt/gitlab/embedded/service/gitlab-rails/lib/feature/definition.rb:174:in load_all!' /opt/gitlab/embedded/service/gitlab-rails/lib/feature/definition.rb:119:in reload!’
/opt/gitlab/embedded/service/gitlab-rails/lib/feature.rb:254:in register_definitions' /opt/gitlab/embedded/service/gitlab-rails/config/initializers/0_inject_feature_flags.rb:7:in <top (required)>’
/opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:7:in <top (required)>' <internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:37:in require’
internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb:37:in require' /opt/gitlab/embedded/bin/bundle:25:in load’
/opt/gitlab/embedded/bin/bundle:25:in <main>' Tasks: TOP => gitlab:db:configure => environment (See full trace by running task with --trace) STDERR: ---- End output of "bash" ---- Ran "bash" returned 1 /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/mixlib-shellout-3.2.7/lib/mixlib/shellout.rb:300:in invalid!’
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/mixlib-shellout-3.2.7/lib/mixlib/shellout.rb:287:in error!' /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/mixlib-shellout-3.2.7/lib/mixlib/shellout/helper.rb:130:in shell_out_compacted!’
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/mixlib-shellout-3.2.7/lib/mixlib/shellout/helper.rb:54:in shell_out!' /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/provider/execute.rb:52:in block (2 levels) in class:Execute
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/mixin/why_run.rb:51:in add_action' /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/provider.rb:293:in converge_by’
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/provider/execute.rb:50:in block in <class:Execute>' (eval):2:in block in action_run’
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/provider.rb:304:in instance_eval' /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/provider.rb:304:in compile_and_converge_action’
(eval):2:in action_run' /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/provider.rb:245:in run_action’
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/resource.rb:601:in block in run_action' /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/resource.rb:628:in with_umask’
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/resource.rb:600:in run_action' /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/runner.rb:74:in run_action’
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/runner.rb:108:in block in run_all_actions' /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/runner.rb:108:in each’
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/runner.rb:108:in run_all_actions' /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/resource_collection.rb:64:in insert’
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/dsl/declare_resource.rb:267:in declare_resource' /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/dsl/resources.rb:36:in bash_hide_env’
/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb:20:in block in class_from_file' (eval):2:in block in action_run’
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/provider.rb:304:in instance_eval' /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/provider.rb:304:in compile_and_converge_action’
(eval):2:in action_run' /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/provider.rb:245:in run_action’
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/resource.rb:601:in block in run_action' /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/resource.rb:628:in with_umask’
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/resource.rb:600:in run_action' /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/runner.rb:74:in run_action’
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/runner.rb:108:in block in run_all_actions' /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/runner.rb:108:in each’
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/runner.rb:108:in run_all_actions' /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/runner.rb:132:in block in converge’
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/resource_collection/resource_list.rb:96:in block in execute_each_resource' /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/resource_collection/stepable_iterator.rb:114:in call_iterator_block’
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/resource_collection/stepable_iterator.rb:85:in step' /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/resource_collection/stepable_iterator.rb:103:in iterate’
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/resource_collection/stepable_iterator.rb:54:in each_with_index' /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/resource_collection/resource_list.rb:94:in execute_each_resource’
/opt/gitlab/embedded/lib/ruby/3.1.0/forwardable.rb:238:in execute_each_resource' /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/runner.rb:130:in converge’
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/client.rb:875:in block in converge' /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/client.rb:870:in catch’
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/client.rb:870:in converge' /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/client.rb:894:in converge_and_save’
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/client.rb:298:in run' /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/application.rb:305:in run_with_graceful_exit_option’
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/application.rb:281:in block in run_chef_client' /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/local_mode.rb:42:in with_server_connectivity’
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/application.rb:264:in run_chef_client' /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/application/base.rb:354:in run_application’
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/application.rb:67:in run' /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-bin-18.3.0/bin/cinc-client:25:in <top (required)>’
/opt/gitlab/embedded/bin/cinc-client:25:in `load’

content in ci_job_artifacts_backlog_large_loop_limit.yml

the errors show the problem in ci_job_artifacts_backlog_large_loop_limit.yml, but I found nothing wrong, the content as follow:

name: ci_job_artifacts_backlog_large_loop_limit
introduced_by_url: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/76509
rollout_issue_url: [Feature flag] Roll out ci_artifact_fast_removal_large_loop_limit (#347151) · Issues · GitLab.org / GitLab · GitLab
milestone: ‘14.10’
type: development
group: group::pipeline security
default_enabled: false