Upgrade to Gitlab v18.5.1 fails with "NoMethodError: undefined method `id' for nil:NilClass"

How to Use GitLab Upgrade
1

Problem to solve

When trying to upgrade my Gitlab-CE Instance v18.4.3 to v18.5.1 (docker compose installation) a DB migration fails and my instance is not starting up.

Any suggestions how I can fix this?

Running handlers:
gitlab  | [2025-10-23T18:33:19+00:00] ERROR: Running exception handlers
gitlab  | There was an error running gitlab-ctl reconfigure:
gitlab  | 
gitlab  | rails_migration[gitlab-rails] (gitlab::database_migrations line 51) had an error: Mixlib::ShellOut::ShellCommandFailed: bash_hide_env[migrate gitlab-rails database] (gitlab::database_migrations line 20) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
gitlab  | ---- Begin output of "bash"  ----
gitlab  | STDOUT: Skipping Topology Service health check due to the cell being disabled
gitlab  | Running db:migrate rake task
gitlab  | main: == [advisory_lock_connection] object_id: 68660, pg_backend_pid: 563
gitlab  | main: == 20250916232115 FinalizeHkFixNonExistingTimelogUsers: migrating =============
gitlab  | main: -- transaction_open?(nil)
gitlab  | main:    -> 0.0000s
gitlab  | main: -- transaction_open?(nil)
gitlab  | main:    -> 0.0000s
gitlab  | main: == [advisory_lock_connection] object_id: 68660, pg_backend_pid: 563
gitlab  | rake aborted!
gitlab  | StandardError: An error has occurred, all later migrations canceled:
gitlab  | 
gitlab  | undefined method `id' for nil:NilClass
gitlab  | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration/fix_non_existing_timelog_users.rb:12:in `perform'
gitlab  | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_wrapper.rb:73:in `execute_batched_migration_job'
gitlab  | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_wrapper.rb:62:in `execute_job'
gitlab  | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_wrapper.rb:50:in `execute_batch'
gitlab  | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_wrapper.rb:25:in `perform'
gitlab  | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_runner.rb:30:in `run_migration_job'
gitlab  | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_runner.rb:146:in `run_migration_while'
gitlab  | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_runner.rb:80:in `finalize'
gitlab  | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_runner.rb:10:in `finalize'
gitlab  | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/batched_background_migration_helpers.rb:161:in `block (2 levels) in finalize_batched_background_migration'
gitlab  | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/query_analyzers/base.rb:52:in `with_suppressed'
gitlab  | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/batched_background_migration_helpers.rb:160:in `block in finalize_batched_background_migration'
gitlab  | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/reestablished_connection_stack.rb:21:in `with_restored_connection_stack'
gitlab  | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/batched_background_migration_helpers.rb:159:in `finalize_batched_background_migration'
gitlab  | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/batched_background_migration_helpers.rb:252:in `ensure_batched_background_migration_is_finished'
gitlab  | /opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20250916232115_finalize_hk_fix_non_existing_timelog_users.rb:11:in `up'
gitlab  | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:33:in `block in exec_migration'
gitlab  | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/query_analyzer.rb:83:in `within'
gitlab  | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:30:in `exec_migration'
gitlab  | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/automatic_lock_writes_on_tables.rb:21:in `exec_migration'
gitlab  | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/require_disable_ddl_transaction_for_multiple_locks.rb:40:in `exec_migration'
gitlab  | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:46:in `ddl_transaction'
gitlab  | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/runner_backoff/active_record_mixin.rb:21:in `execute_migration_in_transaction'
gitlab  | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:14:in `with_advisory_lock'
gitlab  | /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:191:in `configure_database'
gitlab  | Running handlers complete
gitlab  | [2025-10-23T18:33:19+00:00] ERROR: Exception handlers complete
gitlab  | Infra Phase failed. 172 resources updated in 01 minutes 20 seconds
gitlab  | /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:158:in `configure_pg_databases'
gitlab  | /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:102:in `block (3 levels) in <top (required)>'
gitlab  | /opt/gitlab/embedded/bin/bundle:25:in `<main>'
gitlab  | 
gitlab  | Caused by:
gitlab  | NoMethodError: undefined method `id' for nil:NilClass
gitlab  | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration/fix_non_existing_timelog_users.rb:12:in `perform'
gitlab  | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_wrapper.rb:73:in `execute_batched_migration_job'
gitlab  | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_wrapper.rb:62:in `execute_job'
gitlab  | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_wrapper.rb:50:in `execute_batch'
gitlab  | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_wrapper.rb:25:in `perform'
gitlab  | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_runner.rb:30:in `run_migration_job'
gitlab  | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_runner.rb:146:in `run_migration_while'
gitlab  | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_runner.rb:80:in `finalize'
gitlab  | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_runner.rb:10:in `finalize'
gitlab  | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/batched_background_migration_helpers.rb:161:in `block (2 levels) in finalize_batched_background_migration'
gitlab  | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/query_analyzers/base.rb:52:in `with_suppressed'
gitlab  | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/batched_background_migration_helpers.rb:160:in `block in finalize_batched_background_migration'
gitlab  | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/reestablished_connection_stack.rb:21:in `with_restored_connection_stack'
gitlab  | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/batched_background_migration_helpers.rb:159:in `finalize_batched_background_migration'
gitlab  | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/batched_background_migration_helpers.rb:252:in `ensure_batched_background_migration_is_finished'
gitlab  | /opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20250916232115_finalize_hk_fix_non_existing_timelog_users.rb:11:in `up'
gitlab  | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:33:in `block in exec_migration'
gitlab  | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/query_analyzer.rb:83:in `within'
gitlab  | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:30:in `exec_migration'
gitlab  | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/automatic_lock_writes_on_tables.rb:21:in `exec_migration'
gitlab  | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/require_disable_ddl_transaction_for_multiple_locks.rb:40:in `exec_migration'
gitlab  | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:46:in `ddl_transaction'
gitlab  | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/runner_backoff/active_record_mixin.rb:21:in `execute_migration_in_transaction'
gitlab  | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:14:in `with_advisory_lock'
gitlab  | /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:191:in `configure_database'
gitlab  | /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:158:in `configure_pg_databases'
gitlab  | /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:102:in `block (3 levels) in <top (required)>'
gitlab  | /opt/gitlab/embedded/bin/bundle:25:in `<main>'
gitlab  | Tasks: TOP => db:migrate
gitlab  | (See full trace by running task with --trace)
gitlab  | STDERR: 
gitlab  | ---- End output of "bash"  ----
gitlab  | Ran "bash"  returned 1
1 Like
2

Docker exec into the container and check database migration status, and re-run them.

1 Like
3

Unfortunately that’s not possible:
Due to the migrator crashes, it also crashes the container and it restarts in an endless loop.
This means whenever I try to exec into the container with docker compose exec gitlab bash I get kicked out. So I can’t execute statements like gitlab-rake db:migrate:status because there is not enough time until the container crashes again.

Any tips I can prevent the container to restart on failed migration?

4

A container can be run in foreground with an overridden entrypoint, I just don’t remember how the gitlab image behaves (on ipad atm).

Something like

docker run -ti --entrypoint bash gitlab/gitlab-ee

docker compose run --rm --entrypoint bash gitlab

Without the default entrypoint, migrations are not run and you should be able to go into inspection and fix mode manually.

1 Like
5

Thanks for your help. It’s working now, but I’m not sure why.
TL:DR; what I did is:

  • Starting the container with command bash (as you suggested)
  • Start services within container
  • run gitlab-ctl reconfigure inside container.

—-
Step by Step:

STEP 1: I changed the command: setting in my compose.yml like this:

services:
  gitlab:
    image: gitlab/gitlab-ce:18.5.1-ce.0
    container_name: gitlab
    restart: always
    command: [ "bash", "-c", "while true; do sleep 30; done" ]
    ...

STEP 2: Then I opened bash in container with:

docker compose exec gitlab bash

STEP 3: Once inside the container I run these commands to investigate the issue:

/opt/gitlab/embedded/bin/runsvdir /opt/gitlab/service &
# wait here for some time
sleep 5
gitlab-ctl reconfigure
gitlab-ctl status
gitlab-rake db:migrate:status

Everything looks fine.

  • Configuration finishes with success
  • Status reports all up
  • Migrations are all up

Wait until all migrations and background migrations are finished

STEP 4:

  • Exit contrainer and stop it (docker compose down)
  • Remove line command: … from compose.yml again
  • Start container as usual docker compose up -d
1 Like