Unable to restart Gitlab after server reboot

General
1

Hello,

Running Gitlab on CentOS 7, simply restarted the host and gitlab is stuck on 502:

Whoops, GitLab is taking too much time to respond.

The problem seems with gitlab-rake:

# gitlab-rake gitlab:storage:list_legacy_attachments
rake aborted!
NameError: uninitialized constant Arel::Nodes::SqlLiteral
/opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:7:in `<top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => gitlab:storage:list_legacy_attachments => environment
(See full trace by running task with --trace)

Any way to make it work ?
Version: gitlab-ce-15.8.1-ce.0.el7.x86_64

EDIT.

Output with --trace:

# gitlab-rake gitlab:git:fsck --trace
** Invoke gitlab:git:fsck (first_time)
** Invoke gitlab_environment (first_time)
** Execute gitlab_environment
** Invoke environment (first_time)
** Execute environment
rake aborted!
NameError: uninitialized constant Arel::Nodes::SqlLiteral
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.6.1/lib/arel.rb:38:in `sql'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.6.1/lib/active_record/connection_adapters/abstract/database_statements.rb:455:in `<module:DatabaseStatements>'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.6.1/lib/active_record/connection_adapters/abstract/database_statements.rb:5:in `<module:ConnectionAdapters>'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.6.1/lib/active_record/connection_adapters/abstract/database_statements.rb:4:in `<module:ActiveRecord>'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.6.1/lib/active_record/connection_adapters/abstract/database_statements.rb:3:in `<top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/zeitwerk-2.6.6/lib/zeitwerk/kernel.rb:38:in `require'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/zeitwerk-2.6.6/lib/zeitwerk/kernel.rb:38:in `require'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.6.1/lib/active_record/connection_adapters/abstract_adapter.rb:33:in `<class:AbstractAdapter>'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.6.1/lib/active_record/connection_adapters/abstract_adapter.rb:28:in `<module:ConnectionAdapters>'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.6.1/lib/active_record/connection_adapters/abstract_adapter.rb:14:in `<module:ActiveRecord>'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.6.1/lib/active_record/connection_adapters/abstract_adapter.rb:13:in `<top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/zeitwerk-2.6.6/lib/zeitwerk/kernel.rb:38:in `require'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/zeitwerk-2.6.6/lib/zeitwerk/kernel.rb:38:in `require'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.6.1/lib/active_record/connection_adapters/postgresql_adapter.rb:7:in `<top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/zeitwerk-2.6.6/lib/zeitwerk/kernel.rb:38:in `require'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/zeitwerk-2.6.6/lib/zeitwerk/kernel.rb:38:in `require'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.6.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:1205:in `resolve_pool_config'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.6.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:1046:in `establish_connection'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.6.1/lib/active_record/connection_handling.rb:52:in `establish_connection'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.6.1/lib/active_record/railtie.rb:222:in `block (2 levels) in <class:Railtie>'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.1.6.1/lib/active_support/lazy_load_hooks.rb:71:in `class_eval'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.1.6.1/lib/active_support/lazy_load_hooks.rb:71:in `block in execute_hook'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.1.6.1/lib/active_support/lazy_load_hooks.rb:61:in `with_execution_control'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.1.6.1/lib/active_support/lazy_load_hooks.rb:66:in `execute_hook'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.1.6.1/lib/active_support/lazy_load_hooks.rb:43:in `block in on_load'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.1.6.1/lib/active_support/lazy_load_hooks.rb:42:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.1.6.1/lib/active_support/lazy_load_hooks.rb:42:in `on_load'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.6.1/lib/active_record/railtie.rb:217:in `block in <class:Railtie>'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/railties-6.1.6.1/lib/rails/initializable.rb:32:in `instance_exec'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/railties-6.1.6.1/lib/rails/initializable.rb:32:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/railties-6.1.6.1/lib/rails/initializable.rb:61:in `block in run_initializers'
/opt/gitlab/embedded/lib/ruby/2.7.0/tsort.rb:228:in `block in tsort_each'
/opt/gitlab/embedded/lib/ruby/2.7.0/tsort.rb:350:in `block (2 levels) in each_strongly_connected_component'
/opt/gitlab/embedded/lib/ruby/2.7.0/tsort.rb:431:in `each_strongly_connected_component_from'
/opt/gitlab/embedded/lib/ruby/2.7.0/tsort.rb:349:in `block in each_strongly_connected_component'
/opt/gitlab/embedded/lib/ruby/2.7.0/tsort.rb:347:in `each'
/opt/gitlab/embedded/lib/ruby/2.7.0/tsort.rb:347:in `call'
/opt/gitlab/embedded/lib/ruby/2.7.0/tsort.rb:347:in `each_strongly_connected_component'
/opt/gitlab/embedded/lib/ruby/2.7.0/tsort.rb:226:in `tsort_each'
/opt/gitlab/embedded/lib/ruby/2.7.0/tsort.rb:205:in `tsort_each'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/railties-6.1.6.1/lib/rails/initializable.rb:60:in `run_initializers'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/railties-6.1.6.1/lib/rails/application.rb:391:in `initialize!'
/opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:7:in `<top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/zeitwerk-2.6.6/lib/zeitwerk/kernel.rb:38:in `require'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/zeitwerk-2.6.6/lib/zeitwerk/kernel.rb:38:in `require'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/railties-6.1.6.1/lib/rails/application.rb:367:in `require_environment!'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/railties-6.1.6.1/lib/rails/application.rb:533:in `block in run_tasks_blocks'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `block in execute'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `execute'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/sentry-ruby-core-5.1.1/lib/sentry/rake.rb:24:in `execute'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:219:in `block in invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `synchronize'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:188:in `invoke'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/helpers.rake:7:in `block in <top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `block in execute'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `execute'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/sentry-ruby-core-5.1.1/lib/sentry/rake.rb:24:in `execute'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:219:in `block in invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `synchronize'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:243:in `block in invoke_prerequisites'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:241:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:241:in `invoke_prerequisites'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:218:in `block in invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `synchronize'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:188:in `invoke'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:160:in `invoke_task'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `block (2 levels) in top_level'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `block in top_level'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:125:in `run_with_threads'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:110:in `top_level'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:83:in `block in run'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:186:in `standard_exception_handling'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:80:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/exe/rake:27:in `<top (required)>'
/opt/gitlab/embedded/bin/rake:23:in `load'
/opt/gitlab/embedded/bin/rake:23:in `<top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/bundler-2.3.15/lib/bundler/cli/exec.rb:58:in `load'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/bundler-2.3.15/lib/bundler/cli/exec.rb:58:in `kernel_load'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/bundler-2.3.15/lib/bundler/cli/exec.rb:23:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/bundler-2.3.15/lib/bundler/cli.rb:483:in `exec'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/bundler-2.3.15/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/bundler-2.3.15/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/bundler-2.3.15/lib/bundler/vendor/thor/lib/thor.rb:392:in `dispatch'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/bundler-2.3.15/lib/bundler/cli.rb:31:in `dispatch'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/bundler-2.3.15/lib/bundler/vendor/thor/lib/thor/base.rb:485:in `start'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/bundler-2.3.15/lib/bundler/cli.rb:25:in `start'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/bundler-2.3.15/exe/bundle:48:in `block in <top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/bundler-2.3.15/lib/bundler/friendly_errors.rb:117:in `with_friendly_errors'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/bundler-2.3.15/exe/bundle:36:in `<top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => environment
2

Here is supplemental information.
Our nightly backups (/usr/bin/gitlab-rake gitlab:backup:create) where running fine with gitlab-ce-15.7.5.
Then we upgraded to 15.8.1 and it stopped working from there…

Is it a known regression / issue ?

Output the same:

# /usr/bin/gitlab-rake gitlab:backup:create
rake aborted!
NameError: uninitialized constant Arel::Nodes::SqlLiteral
/opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:7:in `<top (required)>'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/helpers.rake:7:in `block in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => environment
(See full trace by running task with --trace)
3

Hello,

We forced a yum reinstall for the current version.

We bypassed some checks by using:
touch /etc/gitlab/skip-unmigrated-data-check
touch /etc/gitlab/skip-auto-backup

Looks like somme migration scripts weren’t fully applied ?

Case closed