Please help fill in this template with all the details to help others help you more efficiently. Use formatting blocks for code, config, logs and ensure to remove sensitive data.
Problem to solve
Omnibus upgrade from 18.2.0 to 18.5.0
after install or gitlab-ctl reconfigure:
================================================================================
Error executing action `run` on resource 'bash_hide_env[migrate gitlab-rails database]'
================================================================================
Mixlib::ShellOut::ShellCommandFailed
------------------------------------
Expected process to exit with [0], but received '1'
---- Begin output of "bash" ----
STDOUT: Skipping Topology Service health check due to the cell being disabled
Running db:migrate rake task
main: == [advisory_lock_connection] object_id: 68660, pg_backend_pid: 2726
main: == 20250916232115 FinalizeHkFixNonExistingTimelogUsers: migrating =============
main: -- transaction_open?(nil)
main: -> 0.0001s
main: -- transaction_open?(nil)
main: -> 0.0000s
main: == [advisory_lock_connection] object_id: 68660, pg_backend_pid: 2726
rake aborted!
StandardError: An error has occurred, all later migrations canceled:
undefined method `id' for nil:NilClass
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration/fix_non_existing_timelog_users.rb:12:in `perform'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_wrapper.rb:73:in `execute_batched_migration_job'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_wrapper.rb:62:in `execute_job'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_wrapper.rb:50:in `execute_batch'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_wrapper.rb:25:in `perform'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_runner.rb:30:in `run_migration_job'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_runner.rb:146:in `run_migration_while'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_runner.rb:80:in `finalize'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_runner.rb:10:in `finalize'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/batched_background_migration_helpers.rb:161:in `block (2 levels) in finalize_batched_background_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/query_analyzers/base.rb:52:in `with_suppressed'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/batched_background_migration_helpers.rb:160:in `block in finalize_batched_background_migration'
...
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:102:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:25:in `<main>'
Caused by:
NoMethodError: undefined method `id' for nil:NilClass
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration/fix_non_existing_timelog_users.rb:12:in `perform'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_wrapper.rb:73:in `execute_batched_migration_job'
...
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:158:in `configure_pg_databases'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:102:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:25:in `<main>'
Tasks: TOP => db:migrate
(See full trace by running task with --trace)
STDERR:
---- End output of "bash" ----
Ran "bash" returned 1
Cookbook Trace: (most recent call first)
----------------------------------------
/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb:20:in `block in class_from_file'
Resource Declaration:
---------------------
# In /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb
20: bash_hide_env "migrate #{new_resource.name} database" do
21: code <<-EOH
22: set -e
23: log_file="#{logging_settings[:log_directory]}/#{new_resource.logfile_prefix}-$(date +%Y-%m-%d-%H-%M-%S).log"
24: umask 077
25: /opt/gitlab/bin/gitlab-rake #{new_resource.rake_task} 2>& 1 | tee ${log_file}
26: STATUS=${PIPESTATUS[0]}
27: chown #{account_helper.gitlab_user}:#{account_helper.gitlab_group} ${log_file}
28: echo $STATUS > #{new_resource.helper.db_migrate_status_file}
29: exit $STATUS
30: EOH
31:
32: environment new_resource.environment if new_resource.property_is_set?(:environment)
33: new_resource.dependent_services.each do |svc|
34: notifies :restart, svc, :immediately
35: end
36:
37: not_if { new_resource.helper.migrated? }
38: sensitive false
39: end
40: end
Compiled Resource:
------------------
# Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb:20:in `block in class_from_file'
bash_hide_env("migrate gitlab-rails database") do
action [:run]
default_guard_interpreter :default
interpreter "bash"
declared_type :bash_hide_env
cookbook_name "gitlab"
recipe_name "database_migrations"
code " set -e\n log_file=\"/var/log/gitlab/gitlab-rails/gitlab-rails-db-migrate-$(date +%Y-%m-%d-%H-%M-%S).log\"\n umask 077\n /opt/gitlab/bin/gitlab-rake gitlab:db:configure 2>& 1 | tee ${log_file}\n STATUS=${PIPESTATUS[0]}\n chown git:git ${log_file}\n echo $STATUS > /var/opt/gitlab/gitlab-rails/upgrade-status/db-migrate-c6697a9cd49e1cfa45cc31b1ecdb90ae-a2f69d15eba\n exit $STATUS\n"
environment "*sensitive value suppressed*"
not_if { #code block }
end
System Info:
------------
chef_version=18.3.0
platform=ubuntu
platform_version=24.04
ruby=ruby 3.2.8 (2025-03-26 revision 13f495dc2c) [x86_64-linux]
program_name=/opt/gitlab/embedded/bin/cinc-client
executable=/opt/gitlab/embedded/bin/cinc-client
================================================================================
Error executing action `run` on resource 'rails_migration[gitlab-rails]'
================================================================================
Mixlib::ShellOut::ShellCommandFailed
------------------------------------
bash_hide_env[migrate gitlab-rails database] (gitlab::database_migrations line 20) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of "bash" ----
STDOUT: Skipping Topology Service health check due to the cell being disabled
Running db:migrate rake task
main: == [advisory_lock_connection] object_id: 68660, pg_backend_pid: 2726
main: == 20250916232115 FinalizeHkFixNonExistingTimelogUsers: migrating =============
main: -- transaction_open?(nil)
main: -> 0.0001s
main: -- transaction_open?(nil)
main: -> 0.0000s
main: == [advisory_lock_connection] object_id: 68660, pg_backend_pid: 2726
rake aborted!
StandardError: An error has occurred, all later migrations canceled:
undefined method `id' for nil:NilClass
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration/fix_non_existing_timelog_users.rb:12:in `perform'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_wrapper.rb:73:in `execute_batched_migration_job'
...
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:102:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:25:in `<main>'
Caused by:
NoMethodError: undefined method `id' for nil:NilClass
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration/fix_non_existing_timelog_users.rb:12:in `perform'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_wrapper.rb:73:in `execute_batched_migration_job'
...
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:158:in `configure_pg_databases'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:102:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:25:in `<main>'
Tasks: TOP => db:migrate
(See full trace by running task with --trace)
STDERR:
---- End output of "bash" ----
Ran "bash" returned 1
Cookbook Trace: (most recent call first)
----------------------------------------
/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb:20:in `block in class_from_file'
Resource Declaration:
---------------------
# In /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb
51: rails_migration "gitlab-rails" do
52: rake_task 'gitlab:db:configure'
53: logfile_prefix 'gitlab-rails-db-migrate'
54: helper migration_helper
55:
56: environment env_variables
57: dependent_services dependent_services
58: notifies :run, "execute[clear the gitlab-rails cache]", :immediately
59: notifies :run, "ruby_block[check remote PG version]", :immediately
60:
61: only_if { migration_helper.attributes_node['auto_migrate'] }
62: end
Compiled Resource:
------------------
# Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb:51:in `from_file'
rails_migration("gitlab-rails") do
action [:run]
default_guard_interpreter :default
declared_type :rails_migration
cookbook_name "gitlab"
recipe_name "database_migrations"
rake_task "gitlab:db:configure"
logfile_prefix "gitlab-rails-db-migrate"
helper "*sensitive value suppressed*"
environment "*sensitive value suppressed*"
dependent_services ["runit_service[puma]", "sidekiq_service[sidekiq]"]
only_if { #code block }
end
System Info:
------------
chef_version=18.3.0
platform=ubuntu
platform_version=24.04
ruby=ruby 3.2.8 (2025-03-26 revision 13f495dc2c) [x86_64-linux]
program_name=/opt/gitlab/embedded/bin/cinc-client
executable=/opt/gitlab/embedded/bin/cinc-client
[2025-11-03T17:42:47+00:00] INFO: Running queued delayed notifications before re-raising exception
Running handlers:
[2025-11-03T17:42:47+00:00] ERROR: Running exception handlers
There was an error running gitlab-ctl reconfigure:
rails_migration[gitlab-rails] (gitlab::database_migrations line 51) had an error: Mixlib::ShellOut::ShellCommandFailed: bash_hide_env[migrate gitlab-rails database] (gitlab::database_migrations line 20) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of "bash" ----
STDOUT: Skipping Topology Service health check due to the cell being disabled
Running db:migrate rake task
main: == [advisory_lock_connection] object_id: 68660, pg_backend_pid: 2726
main: == 20250916232115 FinalizeHkFixNonExistingTimelogUsers: migrating =============
main: -- transaction_open?(nil)
main: -> 0.0001s
main: -- transaction_open?(nil)
main: -> 0.0000s
main: == [advisory_lock_connection] object_id: 68660, pg_backend_pid: 2726
rake aborted!
StandardError: An error has occurred, all later migrations canceled:
undefined method `id' for nil:NilClass
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration/fix_non_existing_timelog_users.rb:12:in `perform'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_wrapper.rb:73:in `execute_batched_migration_job'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_wrapper.rb:62:in `execute_job'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_wrapper.rb:50:in `execute_batch'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_wrapper.rb:25:in `perform'
...
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:102:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:25:in `<main>'
Caused by:
NoMethodError: undefined method `id' for nil:NilClass
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration/fix_non_existing_timelog_users.rb:12:in `perform'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_wrapper.rb:73:in `execute_batched_migration_job'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_wrapper.rb:62:in `execute_job'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_wrapper.rb:50:in `execute_batch'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_wrapper.rb:25:in `perform'
...
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:191:in `configure_database'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:158:in `configure_pg_databases'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:102:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:25:in `<main>'
Tasks: TOP => db:migrate
(See full trace by running task with --trace)
STDERR:
---- End output of "bash" ----
Ran "bash" returned 1
Running handlers complete
[2025-11-03T17:42:47+00:00] ERROR: Exception handlers complete
Infra Phase failed. 3 resources updated in 01 minutes 20 seconds
[2025-11-03T17:42:47+00:00] FATAL: Stacktrace dumped to /opt/gitlab/embedded/cookbooks/cache/cinc-stacktrace.out
[2025-11-03T17:42:47+00:00] FATAL: ---------------------------------------------------------------------------------------
[2025-11-03T17:42:47+00:00] FATAL: PLEASE PROVIDE THE CONTENTS OF THE stacktrace.out FILE (above) IF YOU FILE A BUG REPORT
[2025-11-03T17:42:47+00:00] FATAL: ---------------------------------------------------------------------------------------
[2025-11-03T17:42:47+00:00] FATAL: Mixlib::ShellOut::ShellCommandFailed: rails_migration[gitlab-rails] (gitlab::database_migrations line 51) had an error: Mixlib::ShellOut::ShellCommandFailed: bash_hide_env[migrate gitlab-rails database] (gitlab::database_migrations line 20) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of "bash" ----
STDOUT: Skipping Topology Service health check due to the cell being disabled
Running db:migrate rake task
main: == [advisory_lock_connection] object_id: 68660, pg_backend_pid: 2726
main: == 20250916232115 FinalizeHkFixNonExistingTimelogUsers: migrating =============
main: -- transaction_open?(nil)
main: -> 0.0001s
main: -- transaction_open?(nil)
main: -> 0.0000s
main: == [advisory_lock_connection] object_id: 68660, pg_backend_pid: 2726
rake aborted!
StandardError: An error has occurred, all later migrations canceled:
undefined method `id' for nil:NilClass
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration/fix_non_existing_timelog_users.rb:12:in `perform'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_wrapper.rb:73:in `execute_batched_migration_job'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_wrapper.rb:62:in `execute_job'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_wrapper.rb:50:in `execute_batch'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_wrapper.rb:25:in `perform'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_runner.rb:30:in `run_migration_job'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_runner.rb:146:in `run_migration_while'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_runner.rb:80:in `finalize'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_runner.rb:10:in `finalize'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/batched_background_migration_helpers.rb:161:in `block (2 levels) in finalize_batched_background_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/query_analyzers/base.rb:52:in `with_suppressed'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/batched_background_migration_helpers.rb:160:in `block in finalize_batched_background_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/reestablished_connection_stack.rb:21:in `with_restored_connection_stack'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/batched_background_migration_helpers.rb:159:in `finalize_batched_background_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/batched_background_migration_helpers.rb:252:in `ensure_batched_background_migration_is_finished'
...
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:102:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:25:in `<main>'
Caused by:
NoMethodError: undefined method `id' for nil:NilClass
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration/fix_non_existing_timelog_users.rb:12:in `perform'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_wrapper.rb:73:in `execute_batched_migration_job'
...
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:191:in `configure_database'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:158:in `configure_pg_databases'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:102:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:25:in `<main>'
Tasks: TOP => db:migrate
(See full trace by running task with --trace)
STDERR:
---- End output of "bash" ----
Ran "bash" returned 1
Steps to reproduce
apt install gitlab-ce=18.5.0-ce.0
Configuration
It’s down, so I can’t
Versions
Please add an x whether options apply, and add the version information.
- x Self-managed
-
GitLab.comSaaS - Dedicated
Versions
-
System information
System: Ubuntu 24.04
Current User: git
Using RVM: no
Ruby Version: 3.2.8
Gem Version: 3.7.1
Bundler Version:2.7.1
Rake Version: 13.0.6
Redis Version: 7.2.10
Sidekiq Version:7.3.9
Go Version: unknownGitLab information
Version: 18.5.0
Revision: a2f69d15eba
Directory: /opt/gitlab/embedded/service/gitlab-rails
DB Adapter: PostgreSQL
DB Version: 16.10
URL: https://gitlab.terion.pro
HTTP Clone URL: Sign in · GitLab
SSH Clone URL: git@gitlab.terion.pro:some-group/some-project.git
Using LDAP: no
Using Omniauth: yes
Omniauth Providers:GitLab Shell
Version: 14.45.3
Repository storages:- default: unix:/var/opt/gitlab/gitaly/gitaly.socket
GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell
Gitaly
- default Address: unix:/var/opt/gitlab/gitaly/gitaly.socket
- default Version: 18.5.0
- default Git Version: 2.50.1
- default: unix:/var/opt/gitlab/gitaly/gitaly.socket