Upgrade from 17.8.7 to 17.10.4 failed

ohtizana · April 15, 2025, 10:25pm

Problem to solve

Upgrade failed because:

Running db:migrate rake task
main: == [advisory_lock_connection] object_id: 46800, pg_backend_pid: 603
main: == 20250113163026 FinalizeBackfillCiRunnerMachinesPartitionedTable: migrating =
main: -- transaction_open?(nil)
main:    -> 0.0000s
main: -- transaction_open?(nil)
main:    -> 0.0000s
main: == [advisory_lock_connection] object_id: 46800, pg_backend_pid: 603
STDERR:
Deprecations:
* git_data_dirs has been deprecated since 17.8 and will be removed in 18.0. See https://docs.gitlab.com/omnibus/settings/configuration.html#migrating-from-git_data_dirs for migration instructions.

Update the configuration in your gitlab.rb file or GITLAB_OMNIBUS_CONFIG environment.
---- End output of "bash"  ----
Ran "bash"  returned 1

After add these code to my gitlab.rb, then git-ctl reconfigure, problem remains the same:

gitaly['configuration'] = {
  storage: [
    {
      name: 'default',
      path: '/var/opt/gitlab/git-data/repositories',
    },
  ],
}

Configuration

I am hosting a gitlab community version on my ubuntu server. The gitlab is running as a docker container. I am upgrading from image “gitlab-ce:17.8.7-ce.0” to “gitlab-ce:17.10.4-ce.0”. Both of them are from dockerhub.

Here is my docker compose file:

services:
  gitlab:
    image: gitlab-ce:17.8.7-ce.0   # ---> trying to upgragde it to 17.10.4-ce.0
    container_name: gitlab
    restart: always
    hostname: gitlab.xxxx.com
    environment:
      GITLAB_OMNIBUS_CONFIG: |
        external_url 'http://gitlab.xxxx.com'
    ports:
      - "12000:80"
      - "12001:443"
    volumes:
      - ./gitlab-data/config:/etc/gitlab
      - ./gitlab-data/logs:/var/log/gitlab
      - ./gitlab-data/data:/var/opt/gitlab

Steps to reproduce

docker-compose down
upgrade the image version in docker-compose.yml (first try, 17.8.7-ce.0 → 17.10.4-ce.0)
docker-compose up -d
I saw the error in the log and the documentation from gitlab. The doc says " if your /etc/gitlab/gitlab.rb configuration is as follows, you need to add gitaly[‘configuration’] = xxxx to gtilab.rb"
docker-compose down
change back the image version in docker-compose.yml (17.10.4-ce.0 → 17.8.7-ce.0), otherwise the container keep shutting down and trying restart
I had a look at my ./gitlab-data/config/gitlab.rb, and it’s basically empty — everything is commented out. But I added the gitaly['configuration'] anyway following the doc.
docker-compose up -d
docker exec -it gitlab gitlab-ctl stop
docker exec -it gitlab gitlab-ctl reconfigure
docker exec -it gitlab gitlab-ctl start
docker-compose down
upgrade the image version in docker-compose.yml (second try, 17.8.7-ce.0 → 17.10.4-ce.0)
docker-compose up -d
The same error shows up again.

In my opinion, the document migrating-from-git_data_dirs doesn’t explain what to do if gitlab.rb is empty. That makes me confused.

Versions

Please add an x whether options apply, and add the version information.

Self-managed
GitLab.com SaaS
Dedicated

Versions

GitLab: 17.8.7 to 17.10.4

gamalan · April 16, 2025, 12:32am

happen to us too. but I don’t think it’s because git data dirs. rather, it because this part

 * bash_hide_env[migrate gitlab-rails database] action run
      [execute] rake aborted!
                StandardError: An error has occurred, all later migrations canceled:
                
                uninitialized constant Gitlab::BackgroundMigration::BackfillCiRunnerMachinesPartitionedTable
                /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration.rb:228:in `job_class'
                /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration.rb:157:in `cursor?'
                /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration.rb:214:in `next_min_value'
                /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_runner.rb:103:in `find_next_batch_range'
                /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_runner.rb:94:in `find_or_create_next_batched_job'
                /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_runner.rb:29:in `run_migration_job'
                /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_runner.rb:146:in `run_migration_while'
                /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_runner.rb:80:in `finalize'
                /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_runner.rb:10:in `finalize'
                /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/batched_background_migration_helpers.rb:160:in `block (2 levels) in finalize_batched_background_migration'
                /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/query_analyzers/base.rb:52:in `with_suppressed'
                /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/batched_background_migration_helpers.rb:159:in `block in finalize_batched_background_migration'
                /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/reestablished_connection_stack.rb:21:in `with_restored_connection_stack'
                /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/batched_background_migration_helpers.rb:158:in `finalize_batched_background_migration'
                /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/batched_background_migration_helpers.rb:249:in `ensure_batched_background_migration_is_finished'
                /opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20250113163026_finalize_backfill_ci_runner_machines_partitioned_table.rb:11:in `up'
                /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:33:in `block in exec_migration'
                /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/query_analyzer.rb:83:in `within'
                /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:30:in `exec_migration'
                /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/automatic_lock_writes_on_tables.rb:21:in `exec_migration'
                /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:54:in `ddl_transaction'
                /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/runner_backoff/active_record_mixin.rb:21:in `execute_migration_in_transaction'
                /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:28:in `block in with_advisory_lock_connection'
                /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:25:in `with_advisory_lock_connection'
                /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:145:in `configure_database'
                /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:112:in `configure_pg_databases'
                /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:99:in `block (3 levels) in <top (required)>'
                /opt/gitlab/embedded/bin/bundle:25:in `load'
                /opt/gitlab/embedded/bin/bundle:25:in `<main>'
                
                Caused by:
                NameError: uninitialized constant Gitlab::BackgroundMigration::BackfillCiRunnerMachinesPartitionedTable
                /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration.rb:228:in `job_class'
                /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration.rb:157:in `cursor?'
                /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration.rb:214:in `next_min_value'
                /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_runner.rb:103:in `find_next_batch_range'
                /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_runner.rb:94:in `find_or_create_next_batched_job'
                /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_runner.rb:29:in `run_migration_job'
                /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_runner.rb:146:in `run_migration_while'
                /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_runner.rb:80:in `finalize'
                /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_runner.rb:10:in `finalize'
                /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/batched_background_migration_helpers.rb:160:in `block (2 levels) in finalize_batched_background_migration'
                /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/query_analyzers/base.rb:52:in `with_suppressed'
                /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/batched_background_migration_helpers.rb:159:in `block in finalize_batched_background_migration'
                /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/reestablished_connection_stack.rb:21:in `with_restored_connection_stack'
                /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/batched_background_migration_helpers.rb:158:in `finalize_batched_background_migration'
                /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/batched_background_migration_helpers.rb:249:in `ensure_batched_background_migration_is_finished'
                /opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20250113163026_finalize_backfill_ci_runner_machines_partitioned_table.rb:11:in `up'
                /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:33:in `block in exec_migration'
                /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/query_analyzer.rb:83:in `within'
                /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:30:in `exec_migration'
                /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/automatic_lock_writes_on_tables.rb:21:in `exec_migration'
                /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:54:in `ddl_transaction'
                /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/runner_backoff/active_record_mixin.rb:21:in `execute_migration_in_transaction'
                /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:28:in `block in with_advisory_lock_connection'
                /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:25:in `with_advisory_lock_connection'
                /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:145:in `configure_database'
                /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:112:in `configure_pg_databases'
                /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:99:in `block (3 levels) in <top (required)>'
                /opt/gitlab/embedded/bin/bundle:25:in `load'
                /opt/gitlab/embedded/bin/bundle:25:in `<main>'
                Tasks: TOP => db:migrate
                (See full trace by running task with --trace)
                Running db:migrate rake task
                main: == [advisory_lock_connection] object_id: 48860, pg_backend_pid: 364
                main: == 20250113163026 FinalizeBackfillCiRunnerMachinesPartitionedTable: migrating =
                main: -- transaction_open?(nil)
                main:    -> 0.0000s
                main: -- transaction_open?(nil)
                main:    -> 0.0000s
                main: == [advisory_lock_connection] object_id: 48860, pg_backend_pid: 364

gamalan · April 16, 2025, 1:03am

Testing some workaround in the comment, upgrading to 17.9.3 should help the issue. upgrade to 17.10.3 fails : uninitialized constant Gitlab::BackgroundMigration::BackfillCiRunnerMachinesPartitionedTable (#534087) · Issues · GitLab.org / GitLab · GitLab

ohtizana · April 16, 2025, 5:18am

Thanks @gamalan, it works.

Topic		Replies	Views
Upgrade from an 2+ years old installation of gitlab 6.9 from source with mysql to 8.13 with postgresql with omnibus with docker containers in one shot Upgrade	0	2998	October 26, 2016
Upgrade db migration error from gitlab-ce 7.10.1 to 7.10.4 Upgrade	2	1437	August 23, 2016
Fatal Error of Database when Upgrade GitLab 8.11 to 8.13 Upgrade	0	939	October 26, 2016
Problems Upgrading GitLab in Docker How to Use GitLab docker , upgrade	14	13623	May 26, 2020
GitLab -ctl reconfigure getting aborted after migration from 8.0.4 to 11.7.3 Upgrade docker	0	525	February 19, 2019

Upgrade from 17.8.7 to 17.10.4 failed

Problem to solve

Configuration

Steps to reproduce

Versions

Related topics