RedHat 8.8
Gitlab CE 16.3.6
using dnf to upgrade
I have attempted multiple upgrades past 16.3.6 (16.4, 16.4.9 etc etc) but all fail.
My most recent attempt to go to 16.7.2 fails. I need to rollback the snapshot to get back to 16.3.6. The errors seem to start here. (I have the entire log file of the upgrade attempt
Recipe: gitlab::database_migrations
* ruby_block[check remote PG version] action nothing (skipped due to action :nothing)
* rails_migration[gitlab-rails] action run[2024-01-11T16:00:58-08:00] WARN: gitlab-rails does not have a log_group or default logdir mode defined. Setting to 0700.
* bash_hide_env[migrate gitlab-rails database] action run
[execute] rake aborted!
NameError: uninitialized constant GraphQL::ObjectType
GraphQL::ObjectType.accepts_definitions(authorize: GraphQL::Define.assign_metadata_key(:authorize))
^^^^^^^^^^^^
Did you mean? ObjectSpace
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/graphql.rb:3:in `<top (required)>'
/opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:7:in `<top (required)>'
<internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:38:in `require'
<internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:38:in `require'
/opt/gitlab/embedded/bin/bundle:25:in `load'
/opt/gitlab/embedded/bin/bundle:25:in `<main>'
Tasks: TOP => gitlab:db:configure => environment
(See full trace by running task with --trace)
================================================================================
Error executing action `run` on resource 'bash_hide_env[migrate gitlab-rails database]'
================================================================================
Mixlib::ShellOut::ShellCommandFailed
------------------------------------
Expected process to exit with [0], but received '1'
---- Begin output of "bash" ----
STDOUT: rake aborted!
NameError: uninitialized constant GraphQL::ObjectType
GraphQL::ObjectType.accepts_definitions(authorize: GraphQL::Define.assign_metadata_key(:authorize))
^^^^^^^^^^^^
Did you mean? ObjectSpace
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/graphql.rb:3:in `<top (required)>'
/opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:7:in `<top (required)>'
<internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:38:in `require'
<internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:38:in `require'
/opt/gitlab/embedded/bin/bundle:25:in `load'
/opt/gitlab/embedded/bin/bundle:25:in `<main>'
Tasks: TOP => gitlab:db:configure => environment
(See full trace by running task with --trace)
STDERR:
---- End output of "bash" ----
Ran "bash" returned 1
Cookbook Trace: (most recent call first)
----------------------------------------
/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb:20:in `block in class_from_file'
Resource Declaration:
---------------------
# In /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb
20: bash_hide_env "migrate #{new_resource.name} database" do
21: code <<-EOH
22: set -e
23: log_file="#{logging_settings[:log_directory]}/#{new_resource.logfile_prefix}-$(date +%Y-%m-%d-%H-%M-%S).log"
24: umask 077
25: /opt/gitlab/bin/gitlab-rake #{new_resource.rake_task} 2>& 1 | tee ${log_file}
26: STATUS=${PIPESTATUS[0]}
27: chown #{account_helper.gitlab_user}:#{account_helper.gitlab_group} ${log_file}
28: echo $STATUS > #{new_resource.helper.db_migrate_status_file}
29: exit $STATUS
30: EOH
31:
32: environment new_resource.environment if new_resource.property_is_set?(:environment)
33: new_resource.dependent_services.each do |svc|
34: notifies :restart, svc, :immediately
35: end
36:
37: not_if { new_resource.helper.migrated? }
38: sensitive false
39: end
40: end
Compiled Resource:
------------------
# Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb:20:in `block in class_from_file'
bash_hide_env("migrate gitlab-rails database") do
action [:run]
default_guard_interpreter :default
interpreter "bash"
declared_type :bash_hide_env
cookbook_name "gitlab"
recipe_name "database_migrations"
code " set -e\n log_file=\"/var/log/gitlab/gitlab-rails/gitlab-rails-db-migrate-$(date +%Y-%m-%d-%H-%M-%S).log\"\n umask 077\n /opt/gitlab/bin/gitlab-rake gitlab:db:configure 2>& 1 | tee ${log_file}\n STATUS=${PIPESTATUS[0]}\n chown git:git ${log_file}\n echo $STATUS > /var/opt/gitlab/gitlab-rails/upgrade-status/db-migrate-c6697a9cd49e1cfa45cc31b1ecdb90ae-3c9094d88ea\n exit $STATUS\n"
environment "*sensitive value suppressed*"
sensitive false
domain nil
user nil
not_if { #code block }
end
System Info:
------------
chef_version=17.10.0
platform=redhat
platform_version=8.8
ruby=ruby 3.1.4p223 (2023-03-30 revision 957bb7cb81) [x86_64-linux]
program_name=/opt/gitlab/embedded/bin/cinc-client
executable=/opt/gitlab/embedded/bin/cinc-client
================================================================================
Error executing action `run` on resource 'rails_migration[gitlab-rails]'
================================================================================
Mixlib::ShellOut::ShellCommandFailed
------------------------------------
bash_hide_env[migrate gitlab-rails database] (gitlab::database_migrations line 20) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of "bash" ----
STDOUT: rake aborted!
NameError: uninitialized constant GraphQL::ObjectType
GraphQL::ObjectType.accepts_definitions(authorize: GraphQL::Define.assign_metadata_key(:authorize))
^^^^^^^^^^^^
Did you mean? ObjectSpace
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/graphql.rb:3:in `<top (required)>'
/opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:7:in `<top (required)>'
<internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:38:in `require'
<internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:38:in `require'
/opt/gitlab/embedded/bin/bundle:25:in `load'
/opt/gitlab/embedded/bin/bundle:25:in `<main>'
Tasks: TOP => gitlab:db:configure => environment
(See full trace by running task with --trace)
STDERR:
---- End output of "bash" ----
Ran "bash" returned 1
Cookbook Trace: (most recent call first)
----------------------------------------
/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb:20:in `block in class_from_file'
Resource Declaration:
---------------------
# In /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb
51: rails_migration "gitlab-rails" do
52: rake_task 'gitlab:db:configure'
53: logfile_prefix 'gitlab-rails-db-migrate'
54: helper migration_helper
55:
56: environment env_variables
57: dependent_services dependent_services
58: notifies :run, "execute[clear the gitlab-rails cache]", :immediately
59: notifies :run, "ruby_block[check remote PG version]", :immediately
60:
61: only_if { migration_helper.attributes_node['auto_migrate'] }
62: end
Compiled Resource:
------------------
# Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb:51:in `from_file'
rails_migration("gitlab-rails") do
action [:run]
default_guard_interpreter :default
declared_type :rails_migration
cookbook_name "gitlab"
recipe_name "database_migrations"
rake_task "gitlab:db:configure"
logfile_prefix "gitlab-rails-db-migrate"
helper "*sensitive value suppressed*"
environment "*sensitive value suppressed*"
dependent_services []
only_if { #code block }
end
System Info:
------------
chef_version=17.10.0
platform=redhat
platform_version=8.8
ruby=ruby 3.1.4p223 (2023-03-30 revision 957bb7cb81) [x86_64-linux]
program_name=/opt/gitlab/embedded/bin/cinc-client
executable=/opt/gitlab/embedded/bin/cinc-client
[2024-01-11T16:01:11-08:00] INFO: Running queued delayed notifications before re-raising exception
[2024-01-11T16:01:11-08:00] INFO: templatesymlink[Create a gitlab.yml and create a symlink to Rails root] sending run action to execute[clear the gitlab-rails cache] (delayed)
Recipe: gitlab::gitlab-rails
* execute[clear the gitlab-rails cache] action run
[execute] rake aborted!
NameError: uninitialized constant GraphQL::ObjectType
GraphQL::ObjectType.accepts_definitions(authorize: GraphQL::Define.assign_metadata_key(:authorize))
^^^^^^^^^^^^
Did you mean? ObjectSpace
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/graphql.rb:3:in `<top (required)>'
/opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:7:in `<top (required)>'
<internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:38:in `require'
<internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:38:in `require'
/opt/gitlab/embedded/bin/bundle:25:in `load'
/opt/gitlab/embedded/bin/bundle:25:in `<main>'
Tasks: TOP => cache:clear => cache:clear:redis => environment
(See full trace by running task with --trace)
================================================================================
Error executing action `run` on resource 'execute[clear the gitlab-rails cache]'
================================================================================
Mixlib::ShellOut::ShellCommandFailed
------------------------------------
Expected process to exit with [0], but received '1'
---- Begin output of /opt/gitlab/bin/gitlab-rake cache:clear ----
STDOUT:
STDERR: rake aborted!
NameError: uninitialized constant GraphQL::ObjectType
GraphQL::ObjectType.accepts_definitions(authorize: GraphQL::Define.assign_metadata_key(:authorize))
^^^^^^^^^^^^
Did you mean? ObjectSpace
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/graphql.rb:3:in `<top (required)>'
/opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:7:in `<top (required)>'
<internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:38:in `require'
<internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:38:in `require'
/opt/gitlab/embedded/bin/bundle:25:in `load'
/opt/gitlab/embedded/bin/bundle:25:in `<main>'
Tasks: TOP => cache:clear => cache:clear:redis => environment
(See full trace by running task with --trace)
---- End output of /opt/gitlab/bin/gitlab-rake cache:clear ----
Ran /opt/gitlab/bin/gitlab-rake cache:clear returned 1
Resource Declaration:
---------------------
# In /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/gitlab-rails.rb
554: execute "clear the gitlab-rails cache" do
555: command "/opt/gitlab/bin/gitlab-rake cache:clear"
556: action :nothing
557: not_if { omnibus_helper.not_listening?('redis') || !node['gitlab']['gitlab_rails']['rake_cache_clear'] }
558: end
559:
Compiled Resource:
------------------
# Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/gitlab-rails.rb:554:in `from_file'
execute("clear the gitlab-rails cache") do
action [:nothing]
default_guard_interpreter :execute
command "/opt/gitlab/bin/gitlab-rake cache:clear"
declared_type :execute
cookbook_name "gitlab"
recipe_name "gitlab-rails"
domain nil
user nil
not_if { #code block }
end
System Info:
------------
chef_version=17.10.0
platform=redhat
platform_version=8.8
ruby=ruby 3.1.4p223 (2023-03-30 revision 957bb7cb81) [x86_64-linux]
program_name=/opt/gitlab/embedded/bin/cinc-client
executable=/opt/gitlab/embedded/bin/cinc-client
[2024-01-11T16:01:25-08:00] INFO: version_file[Create version file for Gitaly] sending hup action to runit_service[gitaly] (delayed)
Recipe: gitaly::enable
* runit_service[gitaly] action hup[2024-01-11T16:01:25-08:00] INFO: runit_service[gitaly] signalled (HUP)
[2024-01-11T16:01:25-08:00] INFO: runit_service[gitaly] sent hup
- send hup to runit_service[gitaly]
Running handlers:
[2024-01-11T16:01:25-08:00] ERROR: Running exception handlers
There was an error running gitlab-ctl reconfigure:
Multiple failures occurred:
* Mixlib::ShellOut::ShellCommandFailed occurred in Cinc Client run: rails_migration[gitlab-rails] (gitlab::database_migrations line 51) had an error: Mixlib::ShellOut::ShellCommandFailed: bash_hide_env[migrate gitlab-rails database] (gitlab::database_migrations line 20) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of "bash" ----
STDOUT: rake aborted!
NameError: uninitialized constant GraphQL::ObjectType
GraphQL::ObjectType.accepts_definitions(authorize: GraphQL::Define.assign_metadata_key(:authorize))
^^^^^^^^^^^^
Did you mean? ObjectSpace
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/graphql.rb:3:in `<top (required)>'
/opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:7:in `<top (required)>'
<internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:38:in `require'
<internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:38:in `require'
/opt/gitlab/embedded/bin/bundle:25:in `load'
/opt/gitlab/embedded/bin/bundle:25:in `<main>'
Tasks: TOP => gitlab:db:configure => environment
(See full trace by running task with --trace)
STDERR:
---- End output of "bash" ----
Ran "bash" returned 1
* Mixlib::ShellOut::ShellCommandFailed occurred in delayed notification: execute[clear the gitlab-rails cache] (gitlab::gitlab-rails line 554) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of /opt/gitlab/bin/gitlab-rake cache:clear ----
STDOUT:
STDERR: rake aborted!
NameError: uninitialized constant GraphQL::ObjectType
GraphQL::ObjectType.accepts_definitions(authorize: GraphQL::Define.assign_metadata_key(:authorize))
^^^^^^^^^^^^
Did you mean? ObjectSpace
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/graphql.rb:3:in `<top (required)>'
/opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:7:in `<top (required)>'
<internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:38:in `require'
<internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:38:in `require'
/opt/gitlab/embedded/bin/bundle:25:in `load'
/opt/gitlab/embedded/bin/bundle:25:in `<main>'
Tasks: TOP => cache:clear => cache:clear:redis => environment
(See full trace by running task with --trace)
---- End output of /opt/gitlab/bin/gitlab-rake cache:clear ----
Ran /opt/gitlab/bin/gitlab-rake cache:clear returned 1
Warnings:
The version of the running redis service is different than what is installed.
Please restart redis to start the new version.
sudo gitlab-ctl restart redis
Running handlers complete
[2024-01-11T16:01:25-08:00] ERROR: Exception handlers complete
Infra Phase failed. 29 resources updated in 39 seconds
Warnings:
The version of the running redis service is different than what is installed.
Please restart redis to start the new version.
sudo gitlab-ctl restart redis
[2024-01-11T16:01:25-08:00] FATAL: Stacktrace dumped to /opt/gitlab/embedded/cookbooks/cache/cinc-stacktrace.out
[2024-01-11T16:01:25-08:00] FATAL: ---------------------------------------------------------------------------------------
[2024-01-11T16:01:25-08:00] FATAL: PLEASE PROVIDE THE CONTENTS OF THE stacktrace.out FILE (above) IF YOU FILE A BUG REPORT
[2024-01-11T16:01:25-08:00] FATAL: ---------------------------------------------------------------------------------------
[2024-01-11T16:01:25-08:00] FATAL: Chef::Exceptions::MultipleFailures: Multiple failures occurred:
* Mixlib::ShellOut::ShellCommandFailed occurred in Cinc Client run: rails_migration[gitlab-rails] (gitlab::database_migrations line 51) had an error: Mixlib::ShellOut::ShellCommandFailed: bash_hide_env[migrate gitlab-rails database] (gitlab::database_migrations line 20) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of "bash" ----
STDOUT: rake aborted!
NameError: uninitialized constant GraphQL::ObjectType
GraphQL::ObjectType.accepts_definitions(authorize: GraphQL::Define.assign_metadata_key(:authorize))
^^^^^^^^^^^^
Did you mean? ObjectSpace
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/graphql.rb:3:in `<top (required)>'
/opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:7:in `<top (required)>'
<internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:38:in `require'
<internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:38:in `require'
/opt/gitlab/embedded/bin/bundle:25:in `load'
/opt/gitlab/embedded/bin/bundle:25:in `<main>'
Tasks: TOP => gitlab:db:configure => environment
(See full trace by running task with --trace)
STDERR:
---- End output of "bash" ----
Ran "bash" returned 1
* Mixlib::ShellOut::ShellCommandFailed occurred in delayed notification: execute[clear the gitlab-rails cache] (gitlab::gitlab-rails line 554) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of /opt/gitlab/bin/gitlab-rake cache:clear ----
STDOUT:
STDERR: rake aborted!
NameError: uninitialized constant GraphQL::ObjectType
GraphQL::ObjectType.accepts_definitions(authorize: GraphQL::Define.assign_metadata_key(:authorize))
^^^^^^^^^^^^
Did you mean? ObjectSpace
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/graphql.rb:3:in `<top (required)>'
/opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:7:in `<top (required)>'
<internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:38:in `require'
<internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:38:in `require'
/opt/gitlab/embedded/bin/bundle:25:in `load'
/opt/gitlab/embedded/bin/bundle:25:in `<main>'
Tasks: TOP => cache:clear => cache:clear:redis => environment
(See full trace by running task with --trace)
---- End output of /opt/gitlab/bin/gitlab-rake cache:clear ----
Ran /opt/gitlab/bin/gitlab-rake cache:clear returned 1
===
There was an error running gitlab-ctl reconfigure. Please check the output above for more
details.
===
warning: %posttrans(gitlab-ce-16.7.2-ce.0.el8.x86_64) scriptlet failed, exit status 1
Error in POSTTRANS scriptlet in rpm package gitlab-ce
Verifying : gitlab-ce-16.7.2-ce.0.el8.x86_64 1/2
Verifying : gitlab-ce-16.3.6-ce.0.el8.x86_64 2/2
Installed products updated.
Upgraded:
gitlab-ce-16.7.2-ce.0.el8.x86_64
Complete!