Upgrade from 16.7.3 to 16.9.0 failed

Hello,

I tried to upgrade my GitLab version to 16.9.0 from 16.7.3 but we encountered this error while running the yum install :

46 error: unpacking of archive failed on file /opt/gitlab/embedded/service/mattermost/prepackaged_plugins/mattermost-plugin-servicenow-v2.3.4-linux-amd64.tar.gz;65eb22f3: cpio: read

Since then we cannot do a single command like gitlab-ctl reconfigure or gitlab-rake gitlab:check --trace because the gitlab rake aborts :

** Invoke gitlab:check (first_time)
** Invoke gitlab_environment (first_time)
** Execute gitlab_environment
** Invoke environment (first_time)
** Execute environment
rake aborted!
Feature::InvalidFeatureFlagError: Feature flag 'ci_job_artifacts_backlog_large_loop_limit' is already defined in '/opt/gitlab/embedded/service/gitlab-rails/config/feature_flags/development/ci_job_artifacts_backlog_large_loop_limit.yml'
/opt/gitlab/embedded/service/gitlab-rails/lib/feature/definition.rb:194:in `block in load_all_from_path!'
/opt/gitlab/embedded/service/gitlab-rails/lib/feature/definition.rb:190:in `each'
/opt/gitlab/embedded/service/gitlab-rails/lib/feature/definition.rb:190:in `load_all_from_path!'
/opt/gitlab/embedded/service/gitlab-rails/lib/feature/definition.rb:175:in `block in load_all!'
/opt/gitlab/embedded/service/gitlab-rails/lib/feature/definition.rb:174:in `each'
/opt/gitlab/embedded/service/gitlab-rails/lib/feature/definition.rb:174:in `each_with_object'
/opt/gitlab/embedded/service/gitlab-rails/lib/feature/definition.rb:174:in `load_all!'
/opt/gitlab/embedded/service/gitlab-rails/lib/feature/definition.rb:119:in `reload!'
/opt/gitlab/embedded/service/gitlab-rails/lib/feature.rb:254:in `register_definitions'
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/0_inject_feature_flags.rb:7:in `<top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/railties-7.0.8/lib/rails/engine.rb:667:in `load'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/railties-7.0.8/lib/rails/engine.rb:667:in `block in load_config_initializer'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/activesupport-7.0.8/lib/active_support/notifications.rb:208:in `instrument'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/railties-7.0.8/lib/rails/engine.rb:666:in `load_config_initializer'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/railties-7.0.8/lib/rails/engine.rb:620:in `block (2 levels) in <class:Engine>'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/railties-7.0.8/lib/rails/engine.rb:619:in `each'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/railties-7.0.8/lib/rails/engine.rb:619:in `block in <class:Engine>'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/railties-7.0.8/lib/rails/initializable.rb:32:in `instance_exec'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/railties-7.0.8/lib/rails/initializable.rb:32:in `run'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/railties-7.0.8/lib/rails/initializable.rb:61:in `block in run_initializers'
/opt/gitlab/embedded/lib/ruby/3.1.0/tsort.rb:228:in `block in tsort_each'
/opt/gitlab/embedded/lib/ruby/3.1.0/tsort.rb:350:in `block (2 levels) in each_strongly_connected_component'
/opt/gitlab/embedded/lib/ruby/3.1.0/tsort.rb:422:in `block (2 levels) in each_strongly_connected_component_from'
/opt/gitlab/embedded/lib/ruby/3.1.0/tsort.rb:431:in `each_strongly_connected_component_from'
/opt/gitlab/embedded/lib/ruby/3.1.0/tsort.rb:421:in `block in each_strongly_connected_component_from'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/railties-7.0.8/lib/rails/initializable.rb:50:in `each'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/railties-7.0.8/lib/rails/initializable.rb:50:in `tsort_each_child'
/opt/gitlab/embedded/lib/ruby/3.1.0/tsort.rb:415:in `call'
/opt/gitlab/embedded/lib/ruby/3.1.0/tsort.rb:415:in `each_strongly_connected_component_from'
/opt/gitlab/embedded/lib/ruby/3.1.0/tsort.rb:349:in `block in each_strongly_connected_component'
/opt/gitlab/embedded/lib/ruby/3.1.0/tsort.rb:347:in `each'
/opt/gitlab/embedded/lib/ruby/3.1.0/tsort.rb:347:in `call'
/opt/gitlab/embedded/lib/ruby/3.1.0/tsort.rb:347:in `each_strongly_connected_component'
/opt/gitlab/embedded/lib/ruby/3.1.0/tsort.rb:226:in `tsort_each'
/opt/gitlab/embedded/lib/ruby/3.1.0/tsort.rb:205:in `tsort_each'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/railties-7.0.8/lib/rails/initializable.rb:60:in `run_initializers'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/railties-7.0.8/lib/rails/application.rb:372:in `initialize!'
/opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:7:in `<top (required)>'
<internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:37:in `require'
<internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:37:in `require'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/zeitwerk-2.6.7/lib/zeitwerk/kernel.rb:38:in `require'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/railties-7.0.8/lib/rails/application.rb:348:in `require_environment!'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/railties-7.0.8/lib/rails/application.rb:506:in `block in run_tasks_blocks'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `block in execute'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `each'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `execute'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/sentry-ruby-5.10.0/lib/sentry/rake.rb:24:in `execute'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:219:in `block in invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `synchronize'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:188:in `invoke'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/helpers.rake:7:in `block in <top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `block in execute'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `each'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `execute'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/sentry-ruby-5.10.0/lib/sentry/rake.rb:24:in `execute'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:219:in `block in invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `synchronize'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:243:in `block in invoke_prerequisites'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:241:in `each'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:241:in `invoke_prerequisites'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:218:in `block in invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `synchronize'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:188:in `invoke'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:160:in `invoke_task'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `block (2 levels) in top_level'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `each'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `block in top_level'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:125:in `run_with_threads'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:110:in `top_level'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:83:in `block in run'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:186:in `standard_exception_handling'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:80:in `run'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rake-13.0.6/exe/rake:27:in `<top (required)>'
/opt/gitlab/embedded/bin/rake:25:in `load'
/opt/gitlab/embedded/bin/rake:25:in `<top (required)>'
/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/bundler/cli/exec.rb:58:in `load'
/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/bundler/cli/exec.rb:58:in `kernel_load'
/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/bundler/cli/exec.rb:23:in `run'
/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/bundler/cli.rb:451:in `exec'
/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/bundler/vendor/thor/lib/thor/command.rb:28:in `run'
/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/bundler/vendor/thor/lib/thor.rb:527:in `dispatch'
/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/bundler/cli.rb:34:in `dispatch'
/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/bundler/vendor/thor/lib/thor/base.rb:584:in `start'
/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/bundler/cli.rb:28:in `start'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/bundler-2.5.5/exe/bundle:28:in `block in <top (required)>'
/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/bundler/friendly_errors.rb:117:in `with_friendly_errors'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/bundler-2.5.5/exe/bundle:20:in `<top (required)>'
/opt/gitlab/embedded/bin/bundle:25:in `load'
/opt/gitlab/embedded/bin/bundle:25:in `<main>'
Tasks: TOP => environment

We tried to restore the 16.7.3 with gitlab-backup restore but it fails again because rake aborts.
We also tried to undo with yum but we have an error message.

I feel like I’m stuck and cannot revert or update anymore…

Any help would be appreciated!

Thank you

Were you trying to restore the backup to 16.7.3, or were you trying to restore it to 16.9.0? A backup can only be restored to the version it was taken on, so if you attempted to restore it to 16.9.0 then that won’t work. You would have to reinstall 16.7.3 and restore to that, and then attempt your upgrade again.

Hi iwalker,

The install of 16.9.0 failed so we were still in 16.7.3 version.
I tried to restore from the 16.7.3 backup but the gitlab-rake aborts so it won’t let me restore…

It sounds like you have some stale file that’s defining ci_job_artifacts_backlog_large_loop_limit. Can you search via grep -r ci_job_artifacts_backlog_large_loop_limit /opt/gitlab/embedded/service/gitlab-rails/config/feature_flags?

You may want to rename /opt/gitlab/embedded/service/gitlab-rails to /opt/gitlab/embedded/service/gitlab-rails.old and re-install GitLab to flush out stale files.