GL CE upgrade repeatedly fails. Cannot upgrade past 16.3.6

RedHat 8.8
Gitlab CE 16.3.6
using dnf to upgrade

I have attempted multiple upgrades past 16.3.6 (16.4, 16.4.9 etc etc) but all fail.
My most recent attempt to go to 16.7.2 fails. I need to rollback the snapshot to get back to 16.3.6. The errors seem to start here. (I have the entire log file of the upgrade attempt

Recipe: gitlab::database_migrations
  * ruby_block[check remote PG version] action nothing (skipped due to action :nothing)
  * rails_migration[gitlab-rails] action run[2024-01-11T16:00:58-08:00] WARN: gitlab-rails does not have a log_group or default logdir mode defined. Setting to 0700.

    * bash_hide_env[migrate gitlab-rails database] action run
      [execute] rake aborted!
                NameError: uninitialized constant GraphQL::ObjectType

                GraphQL::ObjectType.accepts_definitions(authorize: GraphQL::Define.assign_metadata_key(:authorize))
                       ^^^^^^^^^^^^
                Did you mean?  ObjectSpace
                /opt/gitlab/embedded/service/gitlab-rails/config/initializers/graphql.rb:3:in `<top (required)>'
                /opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:7:in `<top (required)>'
                <internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:38:in `require'
                <internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:38:in `require'
                /opt/gitlab/embedded/bin/bundle:25:in `load'
                /opt/gitlab/embedded/bin/bundle:25:in `<main>'
                Tasks: TOP => gitlab:db:configure => environment
                (See full trace by running task with --trace)

      ================================================================================
      Error executing action `run` on resource 'bash_hide_env[migrate gitlab-rails database]'
      ================================================================================

      Mixlib::ShellOut::ShellCommandFailed
      ------------------------------------
      Expected process to exit with [0], but received '1'
      ---- Begin output of "bash"  ----
      STDOUT: rake aborted!
      NameError: uninitialized constant GraphQL::ObjectType

      GraphQL::ObjectType.accepts_definitions(authorize: GraphQL::Define.assign_metadata_key(:authorize))
             ^^^^^^^^^^^^
      Did you mean?  ObjectSpace
      /opt/gitlab/embedded/service/gitlab-rails/config/initializers/graphql.rb:3:in `<top (required)>'
      /opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:7:in `<top (required)>'
      <internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:38:in `require'
      <internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:38:in `require'
      /opt/gitlab/embedded/bin/bundle:25:in `load'
      /opt/gitlab/embedded/bin/bundle:25:in `<main>'
      Tasks: TOP => gitlab:db:configure => environment
      (See full trace by running task with --trace)
      STDERR:
      ---- End output of "bash"  ----
      Ran "bash"  returned 1

      Cookbook Trace: (most recent call first)
      ----------------------------------------
      /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb:20:in `block in class_from_file'

      Resource Declaration:
      ---------------------
      # In /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb

       20:   bash_hide_env "migrate #{new_resource.name} database" do
       21:     code <<-EOH
       22:     set -e
       23:     log_file="#{logging_settings[:log_directory]}/#{new_resource.logfile_prefix}-$(date +%Y-%m-%d-%H-%M-%S).log"
       24:     umask 077
       25:     /opt/gitlab/bin/gitlab-rake #{new_resource.rake_task} 2>& 1 | tee ${log_file}
       26:     STATUS=${PIPESTATUS[0]}
       27:     chown #{account_helper.gitlab_user}:#{account_helper.gitlab_group} ${log_file}
       28:     echo $STATUS > #{new_resource.helper.db_migrate_status_file}
       29:     exit $STATUS
       30:     EOH
       31:
       32:     environment new_resource.environment if new_resource.property_is_set?(:environment)
       33:     new_resource.dependent_services.each do |svc|
       34:       notifies :restart, svc, :immediately
       35:     end
       36:
       37:     not_if { new_resource.helper.migrated? }
       38:     sensitive false
       39:   end
       40: end

      Compiled Resource:
      ------------------
      # Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb:20:in `block in class_from_file'

      bash_hide_env("migrate gitlab-rails database") do
        action [:run]
        default_guard_interpreter :default
        interpreter "bash"
        declared_type :bash_hide_env
        cookbook_name "gitlab"
        recipe_name "database_migrations"
        code "    set -e\n    log_file=\"/var/log/gitlab/gitlab-rails/gitlab-rails-db-migrate-$(date +%Y-%m-%d-%H-%M-%S).log\"\n    umask 077\n    /opt/gitlab/bin/gitlab-rake gitlab:db:configure 2>& 1 | tee ${log_file}\n    STATUS=${PIPESTATUS[0]}\n    chown git:git ${log_file}\n    echo $STATUS > /var/opt/gitlab/gitlab-rails/upgrade-status/db-migrate-c6697a9cd49e1cfa45cc31b1ecdb90ae-3c9094d88ea\n    exit $STATUS\n"
        environment "*sensitive value suppressed*"
        sensitive false
        domain nil
        user nil
        not_if { #code block }
      end

      System Info:
      ------------
      chef_version=17.10.0
      platform=redhat
      platform_version=8.8
      ruby=ruby 3.1.4p223 (2023-03-30 revision 957bb7cb81) [x86_64-linux]
      program_name=/opt/gitlab/embedded/bin/cinc-client
      executable=/opt/gitlab/embedded/bin/cinc-client


    ================================================================================
    Error executing action `run` on resource 'rails_migration[gitlab-rails]'
    ================================================================================

    Mixlib::ShellOut::ShellCommandFailed
    ------------------------------------
    bash_hide_env[migrate gitlab-rails database] (gitlab::database_migrations line 20) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
    ---- Begin output of "bash"  ----
    STDOUT: rake aborted!
    NameError: uninitialized constant GraphQL::ObjectType

    GraphQL::ObjectType.accepts_definitions(authorize: GraphQL::Define.assign_metadata_key(:authorize))
           ^^^^^^^^^^^^
    Did you mean?  ObjectSpace
    /opt/gitlab/embedded/service/gitlab-rails/config/initializers/graphql.rb:3:in `<top (required)>'
    /opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:7:in `<top (required)>'
    <internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:38:in `require'
    <internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:38:in `require'
    /opt/gitlab/embedded/bin/bundle:25:in `load'
    /opt/gitlab/embedded/bin/bundle:25:in `<main>'
    Tasks: TOP => gitlab:db:configure => environment
    (See full trace by running task with --trace)
    STDERR:
    ---- End output of "bash"  ----
    Ran "bash"  returned 1

    Cookbook Trace: (most recent call first)
    ----------------------------------------
    /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb:20:in `block in class_from_file'

    Resource Declaration:
    ---------------------
    # In /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb

     51: rails_migration "gitlab-rails" do
     52:   rake_task 'gitlab:db:configure'
     53:   logfile_prefix 'gitlab-rails-db-migrate'
     54:   helper migration_helper
     55:
     56:   environment env_variables
     57:   dependent_services dependent_services
     58:   notifies :run, "execute[clear the gitlab-rails cache]", :immediately
     59:   notifies :run, "ruby_block[check remote PG version]", :immediately
     60:
     61:   only_if { migration_helper.attributes_node['auto_migrate'] }
     62: end

    Compiled Resource:
    ------------------
    # Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb:51:in `from_file'

    rails_migration("gitlab-rails") do
      action [:run]
      default_guard_interpreter :default
      declared_type :rails_migration
      cookbook_name "gitlab"
      recipe_name "database_migrations"
      rake_task "gitlab:db:configure"
      logfile_prefix "gitlab-rails-db-migrate"
      helper "*sensitive value suppressed*"
      environment "*sensitive value suppressed*"
      dependent_services []
      only_if { #code block }
    end

    System Info:
    ------------
    chef_version=17.10.0
    platform=redhat
    platform_version=8.8
    ruby=ruby 3.1.4p223 (2023-03-30 revision 957bb7cb81) [x86_64-linux]
    program_name=/opt/gitlab/embedded/bin/cinc-client
    executable=/opt/gitlab/embedded/bin/cinc-client

[2024-01-11T16:01:11-08:00] INFO: Running queued delayed notifications before re-raising exception
[2024-01-11T16:01:11-08:00] INFO: templatesymlink[Create a gitlab.yml and create a symlink to Rails root] sending run action to execute[clear the gitlab-rails cache] (delayed)
Recipe: gitlab::gitlab-rails
  * execute[clear the gitlab-rails cache] action run
    [execute] rake aborted!
              NameError: uninitialized constant GraphQL::ObjectType

              GraphQL::ObjectType.accepts_definitions(authorize: GraphQL::Define.assign_metadata_key(:authorize))
                     ^^^^^^^^^^^^
              Did you mean?  ObjectSpace
              /opt/gitlab/embedded/service/gitlab-rails/config/initializers/graphql.rb:3:in `<top (required)>'
              /opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:7:in `<top (required)>'
              <internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:38:in `require'
              <internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:38:in `require'
              /opt/gitlab/embedded/bin/bundle:25:in `load'
              /opt/gitlab/embedded/bin/bundle:25:in `<main>'
              Tasks: TOP => cache:clear => cache:clear:redis => environment
              (See full trace by running task with --trace)

    ================================================================================
    Error executing action `run` on resource 'execute[clear the gitlab-rails cache]'
    ================================================================================

    Mixlib::ShellOut::ShellCommandFailed
    ------------------------------------
    Expected process to exit with [0], but received '1'
    ---- Begin output of /opt/gitlab/bin/gitlab-rake cache:clear ----
    STDOUT:
    STDERR: rake aborted!
    NameError: uninitialized constant GraphQL::ObjectType

    GraphQL::ObjectType.accepts_definitions(authorize: GraphQL::Define.assign_metadata_key(:authorize))
           ^^^^^^^^^^^^
    Did you mean?  ObjectSpace
    /opt/gitlab/embedded/service/gitlab-rails/config/initializers/graphql.rb:3:in `<top (required)>'
    /opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:7:in `<top (required)>'
    <internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:38:in `require'
    <internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:38:in `require'
    /opt/gitlab/embedded/bin/bundle:25:in `load'
    /opt/gitlab/embedded/bin/bundle:25:in `<main>'
    Tasks: TOP => cache:clear => cache:clear:redis => environment
    (See full trace by running task with --trace)
    ---- End output of /opt/gitlab/bin/gitlab-rake cache:clear ----
    Ran /opt/gitlab/bin/gitlab-rake cache:clear returned 1

    Resource Declaration:
    ---------------------
    # In /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/gitlab-rails.rb

    554: execute "clear the gitlab-rails cache" do
    555:   command "/opt/gitlab/bin/gitlab-rake cache:clear"
    556:   action :nothing
    557:   not_if { omnibus_helper.not_listening?('redis') || !node['gitlab']['gitlab_rails']['rake_cache_clear'] }
    558: end
    559:

    Compiled Resource:
    ------------------
    # Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/gitlab-rails.rb:554:in `from_file'

    execute("clear the gitlab-rails cache") do
      action [:nothing]
      default_guard_interpreter :execute
      command "/opt/gitlab/bin/gitlab-rake cache:clear"
      declared_type :execute
      cookbook_name "gitlab"
      recipe_name "gitlab-rails"
      domain nil
      user nil
      not_if { #code block }
    end

    System Info:
    ------------
    chef_version=17.10.0
    platform=redhat
    platform_version=8.8
    ruby=ruby 3.1.4p223 (2023-03-30 revision 957bb7cb81) [x86_64-linux]
    program_name=/opt/gitlab/embedded/bin/cinc-client
    executable=/opt/gitlab/embedded/bin/cinc-client

[2024-01-11T16:01:25-08:00] INFO: version_file[Create version file for Gitaly] sending hup action to runit_service[gitaly] (delayed)
Recipe: gitaly::enable
  * runit_service[gitaly] action hup[2024-01-11T16:01:25-08:00] INFO: runit_service[gitaly] signalled (HUP)
[2024-01-11T16:01:25-08:00] INFO: runit_service[gitaly] sent hup

    - send hup to runit_service[gitaly]

Running handlers:
[2024-01-11T16:01:25-08:00] ERROR: Running exception handlers
There was an error running gitlab-ctl reconfigure:

Multiple failures occurred:
* Mixlib::ShellOut::ShellCommandFailed occurred in Cinc Client run: rails_migration[gitlab-rails] (gitlab::database_migrations line 51) had an error: Mixlib::ShellOut::ShellCommandFailed: bash_hide_env[migrate gitlab-rails database] (gitlab::database_migrations line 20) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of "bash"  ----
STDOUT: rake aborted!
NameError: uninitialized constant GraphQL::ObjectType

GraphQL::ObjectType.accepts_definitions(authorize: GraphQL::Define.assign_metadata_key(:authorize))
       ^^^^^^^^^^^^
Did you mean?  ObjectSpace
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/graphql.rb:3:in `<top (required)>'
/opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:7:in `<top (required)>'
<internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:38:in `require'
<internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:38:in `require'
/opt/gitlab/embedded/bin/bundle:25:in `load'
/opt/gitlab/embedded/bin/bundle:25:in `<main>'
Tasks: TOP => gitlab:db:configure => environment
(See full trace by running task with --trace)
STDERR:
---- End output of "bash"  ----
Ran "bash"  returned 1
* Mixlib::ShellOut::ShellCommandFailed occurred in delayed notification: execute[clear the gitlab-rails cache] (gitlab::gitlab-rails line 554) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of /opt/gitlab/bin/gitlab-rake cache:clear ----
STDOUT:
STDERR: rake aborted!
NameError: uninitialized constant GraphQL::ObjectType

GraphQL::ObjectType.accepts_definitions(authorize: GraphQL::Define.assign_metadata_key(:authorize))
       ^^^^^^^^^^^^
Did you mean?  ObjectSpace
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/graphql.rb:3:in `<top (required)>'
/opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:7:in `<top (required)>'
<internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:38:in `require'
<internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:38:in `require'
/opt/gitlab/embedded/bin/bundle:25:in `load'
/opt/gitlab/embedded/bin/bundle:25:in `<main>'
Tasks: TOP => cache:clear => cache:clear:redis => environment
(See full trace by running task with --trace)
---- End output of /opt/gitlab/bin/gitlab-rake cache:clear ----
Ran /opt/gitlab/bin/gitlab-rake cache:clear returned 1


Warnings:
The version of the running redis service is different than what is installed.
Please restart redis to start the new version.

sudo gitlab-ctl restart redis

Running handlers complete
[2024-01-11T16:01:25-08:00] ERROR: Exception handlers complete
Infra Phase failed. 29 resources updated in 39 seconds

Warnings:
The version of the running redis service is different than what is installed.
Please restart redis to start the new version.

sudo gitlab-ctl restart redis

[2024-01-11T16:01:25-08:00] FATAL: Stacktrace dumped to /opt/gitlab/embedded/cookbooks/cache/cinc-stacktrace.out
[2024-01-11T16:01:25-08:00] FATAL: ---------------------------------------------------------------------------------------
[2024-01-11T16:01:25-08:00] FATAL: PLEASE PROVIDE THE CONTENTS OF THE stacktrace.out FILE (above) IF YOU FILE A BUG REPORT
[2024-01-11T16:01:25-08:00] FATAL: ---------------------------------------------------------------------------------------
[2024-01-11T16:01:25-08:00] FATAL: Chef::Exceptions::MultipleFailures: Multiple failures occurred:
* Mixlib::ShellOut::ShellCommandFailed occurred in Cinc Client run: rails_migration[gitlab-rails] (gitlab::database_migrations line 51) had an error: Mixlib::ShellOut::ShellCommandFailed: bash_hide_env[migrate gitlab-rails database] (gitlab::database_migrations line 20) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of "bash"  ----
STDOUT: rake aborted!
NameError: uninitialized constant GraphQL::ObjectType

GraphQL::ObjectType.accepts_definitions(authorize: GraphQL::Define.assign_metadata_key(:authorize))
       ^^^^^^^^^^^^
Did you mean?  ObjectSpace
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/graphql.rb:3:in `<top (required)>'
/opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:7:in `<top (required)>'
<internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:38:in `require'
<internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:38:in `require'
/opt/gitlab/embedded/bin/bundle:25:in `load'
/opt/gitlab/embedded/bin/bundle:25:in `<main>'
Tasks: TOP => gitlab:db:configure => environment
(See full trace by running task with --trace)
STDERR:
---- End output of "bash"  ----
Ran "bash"  returned 1
* Mixlib::ShellOut::ShellCommandFailed occurred in delayed notification: execute[clear the gitlab-rails cache] (gitlab::gitlab-rails line 554) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of /opt/gitlab/bin/gitlab-rake cache:clear ----
STDOUT:
STDERR: rake aborted!
NameError: uninitialized constant GraphQL::ObjectType

GraphQL::ObjectType.accepts_definitions(authorize: GraphQL::Define.assign_metadata_key(:authorize))
       ^^^^^^^^^^^^
Did you mean?  ObjectSpace
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/graphql.rb:3:in `<top (required)>'
/opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:7:in `<top (required)>'
<internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:38:in `require'
<internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:38:in `require'
/opt/gitlab/embedded/bin/bundle:25:in `load'
/opt/gitlab/embedded/bin/bundle:25:in `<main>'
Tasks: TOP => cache:clear => cache:clear:redis => environment
(See full trace by running task with --trace)
---- End output of /opt/gitlab/bin/gitlab-rake cache:clear ----
Ran /opt/gitlab/bin/gitlab-rake cache:clear returned 1

===
There was an error running gitlab-ctl reconfigure. Please check the output above for more
details.
===
warning: %posttrans(gitlab-ce-16.7.2-ce.0.el8.x86_64) scriptlet failed, exit status 1

Error in POSTTRANS scriptlet in rpm package gitlab-ce
  Verifying        : gitlab-ce-16.7.2-ce.0.el8.x86_64                                                               1/2
  Verifying        : gitlab-ce-16.3.6-ce.0.el8.x86_64                                                               2/2
Installed products updated.

Upgraded:
  gitlab-ce-16.7.2-ce.0.el8.x86_64

Complete!

It sounds like there’s an issue with background migrations.

Can you please:

Let us know how it goes!

Sorry gitlab-greg, for the delay. I was out of office. Here is the output you requested:

[foot@gitlab1 ~]$ sudo gitlab-rake db:migrate:status | grep -v up
rake aborted!
NameError: uninitialized constant GraphQL::ObjectType

GraphQL::ObjectType.accepts_definitions(authorize: GraphQL::Define.assign_metadata_key(:authorize))
       ^^^^^^^^^^^^
Did you mean?  ObjectSpace
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/graphql.rb:3:in `<top (required)>'
/opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:7:in `<top (required)>'
<internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:38:in `require'
<internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:38:in `require'
/opt/gitlab/embedded/bin/bundle:25:in `load'
/opt/gitlab/embedded/bin/bundle:25:in `<main>'
Tasks: TOP => db:migrate:status => db:load_config => environment
(See full trace by running task with --trace)

I have the following for upload to you if you require:

[2024-01-30T09:56:20-08:00] FATAL: Stacktrace dumped to /opt/gitlab/embedded/cookbooks/cache/cinc-stacktrace.out
[2024-01-30T09:56:20-08:00] FATAL: ---------------------------------------------------------------------------------------
[2024-01-30T09:56:20-08:00] FATAL: PLEASE PROVIDE THE CONTENTS OF THE stacktrace.out FILE (above) IF YOU FILE A BUG REPORT
[2024-01-30T09:56:20-08:00] FATAL: ---------------------------------------------------------------------------------------

In the meantime, I will roll back the vm snapshot to the functioning 16.3 instance. Thanks

traveling slowed me down, but I’ve updated the issue as you requested. Thanks.

Thanks for the update, @jefilla. There’s definetly something going wrong, but exactly what isn’t quite clear.

output from stacktrace.out:

Generated at 2024-01-30 09:56:20 -0800
Chef::Exceptions::MultipleFailures: Chef::Exceptions::MultipleFailures
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/mixlib-shellout-3.2.7/lib/mixlib/shellout.rb:300:in `invalid!'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/mixlib-shellout-3.2.7/lib/mixlib/shellout.rb:287:in `error!'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/mixlib-shellout-3.2.7/lib/mixlib/shellout/helper.rb:130:in `shell_out_compacted!'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/mixlib-shellout-3.2.7/lib/mixlib/shellout/helper.rb:54:in `shell_out!'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/provider/execute.rb:52:in `block (2 levels) in <class:Execute>'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/mixin/why_run.rb:51:in `add_action'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/provider.rb:293:in `converge_by'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/provider/execute.rb:50:in `block in <class:Execute>'
(eval):2:in `block in action_run'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/provider.rb:304:in `instance_eval'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/provider.rb:304:in `compile_and_converge_action'
(eval):2:in `action_run'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/provider.rb:245:in `run_action'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/resource.rb:601:in `block in run_action'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/resource.rb:628:in `with_umask'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/resource.rb:600:in `run_action'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/runner.rb:74:in `run_action'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/runner.rb:108:in `block in run_all_actions'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/runner.rb:108:in `each'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/runner.rb:108:in `run_all_actions'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/resource_collection.rb:64:in `insert'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/dsl/declare_resource.rb:267:in `declare_resource'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/dsl/resources.rb:36:in `bash_hide_env'
/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb:20:in `block in class_from_file'
(eval):2:in `block in action_run'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/provider.rb:304:in `instance_eval'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/provider.rb:304:in `compile_and_converge_action'
(eval):2:in `action_run'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/provider.rb:245:in `run_action'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/resource.rb:601:in `block in run_action'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/resource.rb:628:in `with_umask'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/resource.rb:600:in `run_action'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/runner.rb:74:in `run_action'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/runner.rb:108:in `block in run_all_actions'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/runner.rb:108:in `each'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/runner.rb:108:in `run_all_actions'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/runner.rb:132:in `block in converge'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/resource_collection/resource_list.rb:96:in `block in execute_each_resource'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/resource_collection/stepable_iterator.rb:114:in `call_iterator_block'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/resource_collection/stepable_iterator.rb:85:in `step'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/resource_collection/stepable_iterator.rb:103:in `iterate'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/resource_collection/stepable_iterator.rb:54:in `each_with_index'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/resource_collection/resource_list.rb:94:in `execute_each_resource'
/opt/gitlab/embedded/lib/ruby/3.1.0/forwardable.rb:238:in `execute_each_resource'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/runner.rb:130:in `converge'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/client.rb:692:in `block in converge'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/client.rb:687:in `catch'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/client.rb:687:in `converge'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/client.rb:711:in `converge_and_save'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/client.rb:285:in `run'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/application.rb:305:in `run_with_graceful_exit_option'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/application.rb:281:in `block in run_chef_client'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/local_mode.rb:42:in `with_server_connectivity'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/application.rb:264:in `run_chef_client'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/application/base.rb:352:in `run_application'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/application.rb:67:in `run'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-bin-17.10.0/bin/cinc-client:25:in `<top (required)>'
/opt/gitlab/embedded/bin/cinc-client:25:in `load'
/opt/gitlab/embedded/bin/cinc-client:25:in `<main>'

>>>> Caused by Mixlib::ShellOut::ShellCommandFailed: rails_migration[gitlab-rails] (gitlab::database_migrations line 51) had an error: Mixlib::ShellOut::ShellCommandFailed: bash_hide_env[migrate gitlab-rails database] (gitlab::database_migrations line 20) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of "bash"  ----
STDOUT: rake aborted!
NameError: uninitialized constant GraphQL::ObjectType

GraphQL::ObjectType.accepts_definitions(authorize: GraphQL::Define.assign_metadata_key(:authorize))
       ^^^^^^^^^^^^
Did you mean?  ObjectSpace
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/graphql.rb:3:in `<top (required)>'
/opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:7:in `<top (required)>'
<internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:38:in `require'
<internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:38:in `require'
/opt/gitlab/embedded/bin/bundle:25:in `load'
/opt/gitlab/embedded/bin/bundle:25:in `<main>'
Tasks: TOP => gitlab:db:configure => environment
(See full trace by running task with --trace)
STDERR:
---- End output of "bash"  ----
Ran "bash"  returned 1
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/mixlib-shellout-3.2.7/lib/mixlib/shellout.rb:300:in `invalid!'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/mixlib-shellout-3.2.7/lib/mixlib/shellout.rb:287:in `error!'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/mixlib-shellout-3.2.7/lib/mixlib/shellout/helper.rb:130:in `shell_out_compacted!'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/mixlib-shellout-3.2.7/lib/mixlib/shellout/helper.rb:54:in `shell_out!'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/provider/execute.rb:52:in `block (2 levels) in <class:Execute>'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/mixin/why_run.rb:51:in `add_action'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/provider.rb:293:in `converge_by'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/provider/execute.rb:50:in `block in <class:Execute>'
(eval):2:in `block in action_run'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/provider.rb:304:in `instance_eval'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/provider.rb:304:in `compile_and_converge_action'
(eval):2:in `action_run'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/provider.rb:245:in `run_action'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/resource.rb:601:in `block in run_action'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/resource.rb:628:in `with_umask'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/resource.rb:600:in `run_action'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/runner.rb:74:in `run_action'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/runner.rb:108:in `block in run_all_actions'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/runner.rb:108:in `each'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/runner.rb:108:in `run_all_actions'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/resource_collection.rb:64:in `insert'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/dsl/declare_resource.rb:267:in `declare_resource'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/dsl/resources.rb:36:in `bash_hide_env'
/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb:20:in `block in class_from_file'
(eval):2:in `block in action_run'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/provider.rb:304:in `instance_eval'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/provider.rb:304:in `compile_and_converge_action'
(eval):2:in `action_run'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/provider.rb:245:in `run_action'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/resource.rb:601:in `block in run_action'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/resource.rb:628:in `with_umask'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/resource.rb:600:in `run_action'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/runner.rb:74:in `run_action'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/runner.rb:108:in `block in run_all_actions'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/runner.rb:108:in `each'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/runner.rb:108:in `run_all_actions'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/runner.rb:132:in `block in converge'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/resource_collection/resource_list.rb:96:in `block in execute_each_resource'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/resource_collection/stepable_iterator.rb:114:in `call_iterator_block'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/resource_collection/stepable_iterator.rb:85:in `step'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/resource_collection/stepable_iterator.rb:103:in `iterate'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/resource_collection/stepable_iterator.rb:54:in `each_with_index'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/resource_collection/resource_list.rb:94:in `execute_each_resource'
/opt/gitlab/embedded/lib/ruby/3.1.0/forwardable.rb:238:in `execute_each_resource'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/runner.rb:130:in `converge'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/client.rb:692:in `block in converge'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/client.rb:687:in `catch'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/client.rb:687:in `converge'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/client.rb:711:in `converge_and_save'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/client.rb:285:in `run'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/application.rb:305:in `run_with_graceful_exit_option'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/application.rb:281:in `block in run_chef_client'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/local_mode.rb:42:in `with_server_connectivity'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/application.rb:264:in `run_chef_client'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/application/base.rb:352:in `run_application'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/application.rb:67:in `run'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-bin-17.10.0/bin/cinc-client:25:in `<top (required)>'
/opt/gitlab/embedded/bin/cinc-client:25:in `load'
/opt/gitlab/embedded/bin/cinc-client:25:in `<main>'

>>>> Caused by Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of "bash"  ----
STDOUT: rake aborted!
NameError: uninitialized constant GraphQL::ObjectType

GraphQL::ObjectType.accepts_definitions(authorize: GraphQL::Define.assign_metadata_key(:authorize))
       ^^^^^^^^^^^^
Did you mean?  ObjectSpace
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/graphql.rb:3:in `<top (required)>'
/opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:7:in `<top (required)>'
<internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:38:in `require'
<internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:38:in `require'
/opt/gitlab/embedded/bin/bundle:25:in `load'
/opt/gitlab/embedded/bin/bundle:25:in `<main>'
Tasks: TOP => gitlab:db:configure => environment
(See full trace by running task with --trace)
STDERR:
---- End output of "bash"  ----
Ran "bash"  returned 1
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/mixlib-shellout-3.2.7/lib/mixlib/shellout.rb:300:in `invalid!'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/mixlib-shellout-3.2.7/lib/mixlib/shellout.rb:287:in `error!'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/mixlib-shellout-3.2.7/lib/mixlib/shellout/helper.rb:130:in `shell_out_compacted!'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/mixlib-shellout-3.2.7/lib/mixlib/shellout/helper.rb:54:in `shell_out!'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/provider/execute.rb:52:in `block (2 levels) in <class:Execute>'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/mixin/why_run.rb:51:in `add_action'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/provider.rb:293:in `converge_by'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/provider/execute.rb:50:in `block in <class:Execute>'
(eval):2:in `block in action_run'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/provider.rb:304:in `instance_eval'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/provider.rb:304:in `compile_and_converge_action'
(eval):2:in `action_run'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/provider.rb:245:in `run_action'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/resource.rb:601:in `block in run_action'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/resource.rb:628:in `with_umask'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/resource.rb:600:in `run_action'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/runner.rb:74:in `run_action'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/runner.rb:108:in `block in run_all_actions'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/runner.rb:108:in `each'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/runner.rb:108:in `run_all_actions'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/resource_collection.rb:64:in `insert'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/dsl/declare_resource.rb:267:in `declare_resource'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/dsl/resources.rb:36:in `bash_hide_env'
/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb:20:in `block in class_from_file'
(eval):2:in `block in action_run'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/provider.rb:304:in `instance_eval'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/provider.rb:304:in `compile_and_converge_action'
(eval):2:in `action_run'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/provider.rb:245:in `run_action'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/resource.rb:601:in `block in run_action'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/resource.rb:628:in `with_umask'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/resource.rb:600:in `run_action'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/runner.rb:74:in `run_action'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/runner.rb:108:in `block in run_all_actions'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/runner.rb:108:in `each'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/runner.rb:108:in `run_all_actions'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/runner.rb:132:in `block in converge'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/resource_collection/resource_list.rb:96:in `block in execute_each_resource'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/resource_collection/stepable_iterator.rb:114:in `call_iterator_block'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/resource_collection/stepable_iterator.rb:85:in `step'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/resource_collection/stepable_iterator.rb:103:in `iterate'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/resource_collection/stepable_iterator.rb:54:in `each_with_index'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/resource_collection/resource_list.rb:94:in `execute_each_resource'
/opt/gitlab/embedded/lib/ruby/3.1.0/forwardable.rb:238:in `execute_each_resource'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/runner.rb:130:in `converge'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/client.rb:692:in `block in converge'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/client.rb:687:in `catch'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/client.rb:687:in `converge'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/client.rb:711:in `converge_and_save'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/client.rb:285:in `run'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/application.rb:305:in `run_with_graceful_exit_option'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/application.rb:281:in `block in run_chef_client'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/local_mode.rb:42:in `with_server_connectivity'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/application.rb:264:in `run_chef_client'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/application/base.rb:352:in `run_application'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-17.10.0/lib/chef/application.rb:67:in `run'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-bin-17.10.0/bin/cinc-client:25:in `<top (required)>'
/opt/gitlab/embedded/bin/cinc-client:25:in `load'
/opt/gitlab/embedded/bin/cinc-client:25:in `<main>'(base) 

from 16.3 instance:

rpm -qa | grep gitlab
gitlab-ce-16.3.6-ce.0.el8.x86_64

sudo gitlab-rake db:migrate:status | grep -v up
GraphQL::ObjectType.accepts_definitions will be removed in GraphQL-Ruby 2.0; use a class-based definition instead. See https://graphql-ruby.org/schema/class_based_api.html.
  -> called from /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/railties-7.0.6/lib/rails/engine.rb:667:in `load'

database: gitlabhq_production

 Status   Migration ID    Migration Name
--------------------------------------------------

16.3 exceptions_log empty

FWIW I unsuccessfully attempted various intermediary upgrades: To 16.4, 16.4.x, 16.6.x. All no go. If I move beyond the current it breaks

Thanks

Thanks for sharing the log entries and additional details. I’ve not seen this issue before.

Can you try re-installing the GitLab 16.3.6-ce package, run gitlab-ctl reconfigure && gitlab-ctl restart and then retry the commands that were failing?

I think either of these command should work to reinstall the package on a EL8 system:

sudo dnf reinstall gitlab-ee-16.3.6-ce
sudo yum reinstall gitlab-ee-16.3.6-ce

Good luck! Let us know how it goes.

Thanks gitlab-greg. I wasn’t clear (“retry the commands that were failing”) if you meant retry an upgrade or gitlab-rake status commands.
I successfully reinstalled the package and after gitlab-ctl reconfigure && gitlab-ctl restart

sudo gitlab-rake db:migrate:status | grep -v up
GraphQL::ObjectType.accepts_definitions will be removed in GraphQL-Ruby 2.0; use a class-based definition instead. See https://graphql-ruby.org/schema/class_based_api.html.
  -> called from /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/railties-7.0.6/lib/rails/engine.rb:667:in `load'

database: gitlabhq_production

 Status   Migration ID    Migration Name
--------------------------------------------------

sudo cat /var/log/gitlab/gitlab-rails/exceptions_json.log
{"severity":"ERROR","time":"2024-02-02T19:57:35.462Z","exception.class":"Redis::CannotConnectError","exception.message":"Error connecting to Redis on /var/opt/gitlab/redis/redis.socket (Errno::ENOENT)","exception.backtrace":["config/initializers/zz_metrics.rb:45:in `connect'","lib/gitlab/instrumentation/redis_interceptor.rb:10:in `block in call'","lib/gitlab/instrumentation/redis_interceptor.rb:41:in `instrument_call'","lib/gitlab/instrumentation/redis_interceptor.rb:9:in `call'"],"exception.cause_class":"Errno::ENOENT","user.username":null,"tags.program":"sidekiq","tags.locale":"en","tags.feature_category":null,"tags.correlation_id":null,"extra.storage":"queues"}
{"severity":"ERROR","time":"2024-02-02T19:57:35.465Z","exception.class":"Redis::CannotConnectError","exception.message":"Error connecting to Redis on /var/opt/gitlab/redis/redis.socket (Errno::ENOENT)","exception.backtrace":["config/initializers/zz_metrics.rb:45:in `connect'","lib/gitlab/instrumentation/redis_interceptor.rb:10:in `block in call'","lib/gitlab/instrumentation/redis_interceptor.rb:41:in `instrument_call'","lib/gitlab/instrumentation/redis_interceptor.rb:9:in `call'"],"exception.cause_class":"Errno::ENOENT","user.username":null,"tags.program":"sidekiq","tags.locale":"en","tags.feature_category":null,"tags.correlation_id":null,"extra.storage":"queues"}
{"severity":"ERROR","time":"2024-02-02T19:57:35.466Z","exception.class":"Redis::CannotConnectError","exception.message":"Error connecting to Redis on /var/opt/gitlab/redis/redis.socket (Errno::ENOENT)","exception.backtrace":["config/initializers/zz_metrics.rb:45:in `connect'","lib/gitlab/instrumentation/redis_interceptor.rb:10:in `block in call'","lib/gitlab/instrumentation/redis_interceptor.rb:41:in `instrument_call'","lib/gitlab/instrumentation/redis_interceptor.rb:9:in `call'"],"exception.cause_class":"Errno::ENOENT","user.username":null,"tags.program":"sidekiq","tags.locale":"en","tags.feature_category":null,"tags.correlation_id":null,"extra.storage":"queues"}
{"severity":"ERROR","time":"2024-02-02T19:57:35.467Z","exception.class":"Redis::CannotConnectError","exception.message":"Error connecting to Redis on /var/opt/gitlab/redis/redis.socket (Errno::ENOENT)","exception.backtrace":["config/initializers/zz_metrics.rb:45:in `connect'","lib/gitlab/instrumentation/redis_interceptor.rb:10:in `block in call'","lib/gitlab/instrumentation/redis_interceptor.rb:41:in `instrument_call'","lib/gitlab/instrumentation/redis_interceptor.rb:9:in `call'"],"exception.cause_class":"Errno::ENOENT","user.username":null,"tags.program":"sidekiq","tags.locale":"en","tags.feature_category":null,"tags.correlation_id":null,"extra.storage":"queues"}
{"severity":"ERROR","time":"2024-02-02T19:57:35.468Z","exception.class":"Redis::CannotConnectError","exception.message":"Error connecting to Redis on /var/opt/gitlab/redis/redis.socket (Errno::ENOENT)","exception.backtrace":["config/initializers/zz_metrics.rb:45:in `connect'","lib/gitlab/instrumentation/redis_interceptor.rb:10:in `block in call'","lib/gitlab/instrumentation/redis_interceptor.rb:41:in `instrument_call'","lib/gitlab/instrumentation/redis_interceptor.rb:9:in `call'"],"exception.cause_class":"Errno::ENOENT","user.username":null,"tags.program":"sidekiq","tags.locale":"en","tags.feature_category":null,"tags.correlation_id":null,"extra.storage":"queues"}
{"severity":"ERROR","time":"2024-02-02T19:57:35.468Z","exception.class":"Redis::CannotConnectError","exception.message":"Error connecting to Redis on /var/opt/gitlab/redis/redis.socket (Errno::ENOENT)","exception.backtrace":["config/initializers/zz_metrics.rb:45:in `connect'","lib/gitlab/instrumentation/redis_interceptor.rb:10:in `block in call'","lib/gitlab/instrumentation/redis_interceptor.rb:41:in `instrument_call'","lib/gitlab/instrumentation/redis_interceptor.rb:9:in `call'"],"exception.cause_class":"Errno::ENOENT","user.username":null,"tags.program":"sidekiq","tags.locale":"en","tags.feature_category":null,"tags.correlation_id":null,"extra.storage":"queues"}
{"severity":"ERROR","time":"2024-02-02T19:57:35.469Z","exception.class":"Redis::CannotConnectError","exception.message":"Error connecting to Redis on /var/opt/gitlab/redis/redis.socket (Errno::ENOENT)","exception.backtrace":["config/initializers/zz_metrics.rb:45:in `connect'","lib/gitlab/instrumentation/redis_interceptor.rb:10:in `block in call'","lib/gitlab/instrumentation/redis_interceptor.rb:41:in `instrument_call'","lib/gitlab/instrumentation/redis_interceptor.rb:9:in `call'"],"exception.cause_class":"Errno::ENOENT","user.username":null,"tags.program":"sidekiq","tags.locale":"en","tags.feature_category":null,"tags.correlation_id":null,"extra.storage":"queues"}
{"severity":"ERROR","time":"2024-02-02T19:57:35.470Z","exception.class":"Redis::CannotConnectError","exception.message":"Error connecting to Redis on /var/opt/gitlab/redis/redis.socket (Errno::ENOENT)","exception.backtrace":["config/initializers/zz_metrics.rb:45:in `connect'","lib/gitlab/instrumentation/redis_interceptor.rb:10:in `block in call'","lib/gitlab/instrumentation/redis_interceptor.rb:41:in `instrument_call'","lib/gitlab/instrumentation/redis_interceptor.rb:9:in `call'"],"exception.cause_class":"Errno::ENOENT","user.username":null,"tags.program":"sidekiq","tags.locale":"en","tags.feature_category":null,"tags.correlation_id":null,"extra.storage":"queues"}
{"severity":"ERROR","time":"2024-02-02T19:57:35.470Z","exception.class":"Redis::CannotConnectError","exception.message":"Error connecting to Redis on /var/opt/gitlab/redis/redis.socket (Errno::ENOENT)","exception.backtrace":["config/initializers/zz_metrics.rb:45:in `connect'","lib/gitlab/instrumentation/redis_interceptor.rb:10:in `block in call'","lib/gitlab/instrumentation/redis_interceptor.rb:41:in `instrument_call'","lib/gitlab/instrumentation/redis_interceptor.rb:9:in `call'"],"exception.cause_class":"Errno::ENOENT","user.username":null,"tags.program":"sidekiq","tags.locale":"en","tags.feature_category":null,"tags.correlation_id":null,"extra.storage":"queues"}
{"severity":"ERROR","time":"2024-02-02T19:57:35.471Z","exception.class":"Redis::CannotConnectError","exception.message":"Error connecting to Redis on /var/opt/gitlab/redis/redis.socket (Errno::ENOENT)","exception.backtrace":["config/initializers/zz_metrics.rb:45:in `connect'","lib/gitlab/instrumentation/redis_interceptor.rb:10:in `block in call'","lib/gitlab/instrumentation/redis_interceptor.rb:41:in `instrument_call'","lib/gitlab/instrumentation/redis_interceptor.rb:9:in `call'"],"exception.cause_class":"Errno::ENOENT","user.username":null,"tags.program":"sidekiq","tags.locale":"en","tags.feature_category":null,"tags.correlation_id":null,"extra.storage":"queues"}
{"severity":"ERROR","time":"2024-02-02T19:57:35.472Z","exception.class":"Redis::CannotConnectError","exception.message":"Error connecting to Redis on /var/opt/gitlab/redis/redis.socket (Errno::ENOENT)","exception.backtrace":["config/initializers/zz_metrics.rb:45:in `connect'","lib/gitlab/instrumentation/redis_interceptor.rb:10:in `block in call'","lib/gitlab/instrumentation/redis_interceptor.rb:41:in `instrument_call'","lib/gitlab/instrumentation/redis_interceptor.rb:9:in `call'"],"exception.cause_class":"Errno::ENOENT","user.username":null,"tags.program":"sidekiq","tags.locale":"en","tags.feature_category":null,"tags.correlation_id":null,"extra.storage":"queues"}
{"severity":"ERROR","time":"2024-02-02T19:57:35.472Z","exception.class":"Redis::CannotConnectError","exception.message":"Error connecting to Redis on /var/opt/gitlab/redis/redis.socket (Errno::ENOENT)","exception.backtrace":["config/initializers/zz_metrics.rb:45:in `connect'","lib/gitlab/instrumentation/redis_interceptor.rb:10:in `block in call'","lib/gitlab/instrumentation/redis_interceptor.rb:41:in `instrument_call'","lib/gitlab/instrumentation/redis_interceptor.rb:9:in `call'"],"exception.cause_class":"Errno::ENOENT","user.username":null,"tags.program":"sidekiq","tags.locale":"en","tags.feature_category":null,"tags.correlation_id":null,"extra.storage":"queues"}
{"severity":"ERROR","time":"2024-02-02T19:57:35.473Z","exception.class":"Redis::CannotConnectError","exception.message":"Error connecting to Redis on /var/opt/gitlab/redis/redis.socket (Errno::ENOENT)","exception.backtrace":["config/initializers/zz_metrics.rb:45:in `connect'","lib/gitlab/instrumentation/redis_interceptor.rb:10:in `block in call'","lib/gitlab/instrumentation/redis_interceptor.rb:41:in `instrument_call'","lib/gitlab/instrumentation/redis_interceptor.rb:9:in `call'"],"exception.cause_class":"Errno::ENOENT","user.username":null,"tags.program":"sidekiq","tags.locale":"en","tags.feature_category":null,"tags.correlation_id":null,"extra.storage":"queues"}
{"severity":"ERROR","time":"2024-02-02T19:57:35.474Z","exception.class":"Redis::CannotConnectError","exception.message":"Error connecting to Redis on /var/opt/gitlab/redis/redis.socket (Errno::ENOENT)","exception.backtrace":["config/initializers/zz_metrics.rb:45:in `connect'","lib/gitlab/instrumentation/redis_interceptor.rb:10:in `block in call'","lib/gitlab/instrumentation/redis_interceptor.rb:41:in `instrument_call'","lib/gitlab/instrumentation/redis_interceptor.rb:9:in `call'"],"exception.cause_class":"Errno::ENOENT","user.username":null,"tags.program":"sidekiq","tags.locale":"en","tags.feature_category":null,"tags.correlation_id":null,"extra.storage":"queues"}
{"severity":"ERROR","time":"2024-02-02T19:57:35.475Z","exception.class":"Redis::CannotConnectError","exception.message":"Error connecting to Redis on /var/opt/gitlab/redis/redis.socket (Errno::ENOENT)","exception.backtrace":["config/initializers/zz_metrics.rb:45:in `connect'","lib/gitlab/instrumentation/redis_interceptor.rb:10:in `block in call'","lib/gitlab/instrumentation/redis_interceptor.rb:41:in `instrument_call'","lib/gitlab/instrumentation/redis_interceptor.rb:9:in `call'"],"exception.cause_class":"Errno::ENOENT","user.username":null,"tags.program":"sidekiq","tags.locale":"en","tags.feature_category":null,"tags.correlation_id":null,"extra.storage":"queues"}
{"severity":"ERROR","time":"2024-02-02T19:57:35.477Z","exception.class":"Redis::CannotConnectError","exception.message":"Error connecting to Redis on /var/opt/gitlab/redis/redis.socket (Errno::ENOENT)","exception.backtrace":["config/initializers/zz_metrics.rb:45:in `connect'","lib/gitlab/instrumentation/redis_interceptor.rb:10:in `block in call'","lib/gitlab/instrumentation/redis_interceptor.rb:41:in `instrument_call'","lib/gitlab/instrumentation/redis_interceptor.rb:9:in `call'"],"exception.cause_class":"Errno::ENOENT","user.username":null,"tags.program":"sidekiq","tags.locale":"en","tags.feature_category":null,"tags.correlation_id":null,"extra.storage":"queues"}
{"severity":"ERROR","time":"2024-02-02T19:57:35.478Z","exception.class":"Redis::CannotConnectError","exception.message":"Error connecting to Redis on /var/opt/gitlab/redis/redis.socket (Errno::ENOENT)","exception.backtrace":["config/initializers/zz_metrics.rb:45:in `connect'","lib/gitlab/instrumentation/redis_interceptor.rb:10:in `block in call'","lib/gitlab/instrumentation/redis_interceptor.rb:41:in `instrument_call'","lib/gitlab/instrumentation/redis_interceptor.rb:9:in `call'"],"exception.cause_class":"Errno::ENOENT","user.username":null,"tags.program":"sidekiq","tags.locale":"en","tags.feature_category":null,"tags.correlation_id":null,"extra.storage":"queues"}
{"severity":"ERROR","time":"2024-02-02T19:57:35.479Z","exception.class":"Redis::CannotConnectError","exception.message":"Error connecting to Redis on /var/opt/gitlab/redis/redis.socket (Errno::ENOENT)","exception.backtrace":["config/initializers/zz_metrics.rb:45:in `connect'","lib/gitlab/instrumentation/redis_interceptor.rb:10:in `block in call'","lib/gitlab/instrumentation/redis_interceptor.rb:41:in `instrument_call'","lib/gitlab/instrumentation/redis_interceptor.rb:9:in `call'"],"exception.cause_class":"Errno::ENOENT","user.username":null,"tags.program":"sidekiq","tags.locale":"en","tags.feature_category":null,"tags.correlation_id":null,"extra.storage":"queues"}
{"severity":"ERROR","time":"2024-02-02T19:57:35.480Z","exception.class":"Redis::CannotConnectError","exception.message":"Error connecting to Redis on /var/opt/gitlab/redis/redis.socket (Errno::ENOENT)","exception.backtrace":["config/initializers/zz_metrics.rb:45:in `connect'","lib/gitlab/instrumentation/redis_interceptor.rb:10:in `block in call'","lib/gitlab/instrumentation/redis_interceptor.rb:41:in `instrument_call'","lib/gitlab/instrumentation/redis_interceptor.rb:9:in `call'"],"exception.cause_class":"Errno::ENOENT","user.username":null,"tags.program":"sidekiq","tags.locale":"en","tags.feature_category":null,"tags.correlation_id":null,"extra.storage":"queues"}
{"severity":"ERROR","time":"2024-02-02T19:57:35.480Z","exception.class":"Redis::CannotConnectError","exception.message":"Error connecting to Redis on /var/opt/gitlab/redis/redis.socket (Errno::ENOENT)","exception.backtrace":["config/initializers/zz_metrics.rb:45:in `connect'","lib/gitlab/instrumentation/redis_interceptor.rb:10:in `block in call'","lib/gitlab/instrumentation/redis_interceptor.rb:41:in `instrument_call'","lib/gitlab/instrumentation/redis_interceptor.rb:9:in `call'"],"exception.cause_class":"Errno::ENOENT","user.username":null,"tags.program":"sidekiq","tags.locale":"en","tags.feature_category":null,"tags.correlation_id":null,"extra.storage":"queues"}

Thank you for your help and patience. I’ll wait to hear any feedback before attempting any upgrades