GitLab 13.12.5 db:migrate error

cdenneen · July 1, 2021, 5:43pm

** Invoke db:migrate (first_time)
** Invoke db:load_config (first_time)
** Invoke environment (first_time)
** Execute environment
** Execute db:load_config
** Invoke schema_version_check (first_time)
** Invoke environment
** Execute schema_version_check
** Execute db:migrate
== 20210306121310 BackfillPartitionedWebHookLogs: migrating ===================
-- transaction_open?()
   -> 0.0001s
WARNING: Active Record does not support composite primary key.

web_hook_logs has composite primary key. Composite primary key is ignored.
rails aborted!
StandardError: An error has occurred, all later migrations canceled:

the column: argument must be set to a column name to use for ordering rows
/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/each_batch.rb:52:in `each_batch'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/background_migration_helpers.rb:109:in `queue_background_migration_jobs_by_range_at_intervals'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/partitioning_migration_helpers/table_management_helpers.rb:395:in `enqueue_background_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/partitioning_migration_helpers/table_management_helpers.rb:108:in `enqueue_partitioning_data_migration'
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20210306121310_backfill_partitioned_web_hook_logs.rb:11:in `up'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.0.3.6/lib/active_record/migration.rb:831:in `exec_migration'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.0.3.6/lib/active_record/migration.rb:812:in `block (2 levels) in migrate'
/opt/gitlab/embedded/lib/ruby/2.7.0/benchmark.rb:293:in `measure'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.0.3.6/lib/active_record/migration.rb:811:in `block in migrate'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.0.3.6/lib/active_record/connection_adapters/abstract/connection_pool.rb:471:in `with_connection'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.0.3.6/lib/active_record/migration.rb:810:in `migrate'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.0.3.6/lib/active_record/migration.rb:1002:in `migrate'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.0.3.6/lib/active_record/migration.rb:1310:in `block in execute_migration_in_transaction'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.0.3.6/lib/active_record/migration.rb:1363:in `ddl_transaction'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.0.3.6/lib/active_record/migration.rb:1309:in `execute_migration_in_transaction'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.0.3.6/lib/active_record/migration.rb:1281:in `block in migrate_without_lock'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.0.3.6/lib/active_record/migration.rb:1280:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.0.3.6/lib/active_record/migration.rb:1280:in `migrate_without_lock'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.0.3.6/lib/active_record/migration.rb:1229:in `block in migrate'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.0.3.6/lib/active_record/migration.rb:1382:in `with_advisory_lock'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.0.3.6/lib/active_record/migration.rb:1229:in `migrate'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.0.3.6/lib/active_record/migration.rb:1061:in `up'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.0.3.6/lib/active_record/migration.rb:1036:in `migrate'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.0.3.6/lib/active_record/tasks/database_tasks.rb:238:in `migrate'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.0.3.6/lib/active_record/railties/databases.rake:86:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.0.3.6/lib/active_record/railties/databases.rake:84:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.0.3.6/lib/active_record/railties/databases.rake:84:in `block (2 levels) in <top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.3/lib/rake/task.rb:281:in `block in execute'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.3/lib/rake/task.rb:281:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.3/lib/rake/task.rb:281:in `execute'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.3/lib/rake/task.rb:219:in `block in invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.3/lib/rake/task.rb:199:in `synchronize'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.3/lib/rake/task.rb:199:in `invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.3/lib/rake/task.rb:188:in `invoke'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.3/lib/rake/application.rb:160:in `invoke_task'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.3/lib/rake/application.rb:116:in `block (2 levels) in top_level'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.3/lib/rake/application.rb:116:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.3/lib/rake/application.rb:116:in `block in top_level'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.3/lib/rake/application.rb:125:in `run_with_threads'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.3/lib/rake/application.rb:110:in `top_level'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/railties-6.0.3.6/lib/rails/commands/rake/rake_command.rb:23:in `block in perform'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.3/lib/rake/application.rb:186:in `standard_exception_handling'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/railties-6.0.3.6/lib/rails/commands/rake/rake_command.rb:20:in `perform'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/railties-6.0.3.6/lib/rails/command.rb:48:in `invoke'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/railties-6.0.3.6/lib/rails/commands.rb:18:in `<top (required)>'
bin/rails:4:in `require'
bin/rails:4:in `<main>'

Caused by:
ArgumentError: the column: argument must be set to a column name to use for ordering rows
/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/each_batch.rb:52:in `each_batch'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/background_migration_helpers.rb:109:in `queue_background_migration_jobs_by_range_at_intervals'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/partitioning_migration_helpers/table_management_helpers.rb:395:in `enqueue_background_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/partitioning_migration_helpers/table_management_helpers.rb:108:in `enqueue_partitioning_data_migration'
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20210306121310_backfill_partitioned_web_hook_logs.rb:11:in `up'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.0.3.6/lib/active_record/migration.rb:831:in `exec_migration'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.0.3.6/lib/active_record/migration.rb:812:in `block (2 levels) in migrate'
/opt/gitlab/embedded/lib/ruby/2.7.0/benchmark.rb:293:in `measure'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.0.3.6/lib/active_record/migration.rb:811:in `block in migrate'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.0.3.6/lib/active_record/connection_adapters/abstract/connection_pool.rb:471:in `with_connection'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.0.3.6/lib/active_record/migration.rb:810:in `migrate'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.0.3.6/lib/active_record/migration.rb:1002:in `migrate'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.0.3.6/lib/active_record/migration.rb:1310:in `block in execute_migration_in_transaction'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.0.3.6/lib/active_record/migration.rb:1363:in `ddl_transaction'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.0.3.6/lib/active_record/migration.rb:1309:in `execute_migration_in_transaction'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.0.3.6/lib/active_record/migration.rb:1281:in `block in migrate_without_lock'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.0.3.6/lib/active_record/migration.rb:1280:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.0.3.6/lib/active_record/migration.rb:1280:in `migrate_without_lock'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.0.3.6/lib/active_record/migration.rb:1229:in `block in migrate'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.0.3.6/lib/active_record/migration.rb:1382:in `with_advisory_lock'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.0.3.6/lib/active_record/migration.rb:1229:in `migrate'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.0.3.6/lib/active_record/migration.rb:1061:in `up'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.0.3.6/lib/active_record/migration.rb:1036:in `migrate'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.0.3.6/lib/active_record/tasks/database_tasks.rb:238:in `migrate'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.0.3.6/lib/active_record/railties/databases.rake:86:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.0.3.6/lib/active_record/railties/databases.rake:84:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.0.3.6/lib/active_record/railties/databases.rake:84:in `block (2 levels) in <top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.3/lib/rake/task.rb:281:in `block in execute'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.3/lib/rake/task.rb:281:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.3/lib/rake/task.rb:281:in `execute'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.3/lib/rake/task.rb:219:in `block in invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.3/lib/rake/task.rb:199:in `synchronize'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.3/lib/rake/task.rb:199:in `invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.3/lib/rake/task.rb:188:in `invoke'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.3/lib/rake/application.rb:160:in `invoke_task'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.3/lib/rake/application.rb:116:in `block (2 levels) in top_level'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.3/lib/rake/application.rb:116:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.3/lib/rake/application.rb:116:in `block in top_level'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.3/lib/rake/application.rb:125:in `run_with_threads'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.3/lib/rake/application.rb:110:in `top_level'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/railties-6.0.3.6/lib/rails/commands/rake/rake_command.rb:23:in `block in perform'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.3/lib/rake/application.rb:186:in `standard_exception_handling'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/railties-6.0.3.6/lib/rails/commands/rake/rake_command.rb:20:in `perform'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/railties-6.0.3.6/lib/rails/command.rb:48:in `invoke'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/railties-6.0.3.6/lib/rails/commands.rb:18:in `<top (required)>'
bin/rails:4:in `require'
bin/rails:4:in `<main>'
Tasks: TOP => db:migrate
root@ip-10-208-43-120:~#

Found this but it didn’t work: Omnibus Upgrade from 13.11.4 causing the issue PG::UndefinedObject: ERROR: constraint “web_hook_logs_pkey” - How to Use GitLab / Upgrade - GitLab Forum

Trying to do so yielded:

gitlabhq_production=> ALTER TABLE web_hook_logs ADD PRIMARY KEY USING INDEX web_hook_logs_pkey;
ERROR:  index "web_hook_logs_pkey" is already associated with a constraint
LINE 1: ALTER TABLE web_hook_logs ADD PRIMARY KEY USING INDEX web_ho...

kzli · July 11, 2021, 7:03am

When I update from 13.5.4 → 13.12.8.

–
WARNING: Active Record does not support composite primary key.

web_hook_logs has composite primary key. Composite primary key is ignored.

rake aborted!

StandardError: An error has occurred, all later migrations canceled:

the column: argument must be set to a column name to use for ordering rows

/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/each_batch.rb:52:in `each_batch’

/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/background_migration_helpers.rb:109:in `queue_background_migration_jobs_by_range_at_intervals’

/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/partitioning_migration_helpers/table_management_helpers.rb:395:in `enqueue_background_migration’

/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/partitioning_migration_helpers/table_management_helpers.rb:108:in `enqueue_partitioning_data_migration’

/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20210306121310_backfill_partitioned_web_hook_logs.rb:11:in `up’

/opt/gitlab/embedded/bin/bundle:23:in `load’

/opt/gitlab/embedded/bin/bundle:23:in `’

Caused by:

ArgumentError: the column: argument must be set to a column name to use for ordering rows

/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/each_batch.rb:52:in `each_batch’

/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/background_migration_helpers.rb:109:in `queue_background_migration_jobs_by_range_at_intervals’

/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/partitioning_migration_helpers/table_management_helpers.rb:395:in `enqueue_background_migration’

/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/partitioning_migration_helpers/table_management_helpers.rb:108:in `enqueue_partitioning_data_migration’

/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20210306121310_backfill_partitioned_web_hook_logs.rb:11:in `up’

/opt/gitlab/embedded/bin/bundle:23:in `load’

/opt/gitlab/embedded/bin/bundle:23:in `’

Tasks: TOP => db:migrate

(See full trace by running task with --trace)

Enigma · July 12, 2021, 3:07pm

I ran into the same issue after migrations began to fail unknowingly after a prior migration bug with no-downtime upgrades, which caused some migrations to not be performed across minor versions.

Based on my research, this issue occurs if you perform a no-downtime upgrade across multiple minor versions to 13.12, or in my case, the problematic migration (20210306121310 BackfillPartitionedWebHookLogs) was not performed due to a prior failing migration. The reason is GitLab only supports no downtime upgrades across a single minor version due to the existence of pre- and post-upgrade migrations, where pre migrations are run before the new software is installed and post migrations are run afterwards. Later minor versions may contain pre migrations that break older post migrations, which only poses an issue for no-downtime upgrades across multiple minor versions.

Per this issue, the solution is to revert the later pre migration, run the older post migrations that depend on the changed table, then re-run the pre migration:

gitlab-rake db:migrate:down VERSION=20210424115725
gitlab-rake db:migrate:up VERSION=20210306121310
gitlab-rake db:migrate:up VERSION=20210326121537
gitlab-rake db:migrate:up VERSION=20210413130011
gitlab-rake db:migrate:up VERSION=20210424115725

cdenneen · July 12, 2021, 4:22pm

I opened a GitLab issue on this and already tried those steps with no success:

Upgrade 13.12.5 from 13.7.9 has db:migrate error (#335026) · Issues · GitLab.org / GitLab · GitLab

Topic		Replies	Views
Conversion from CE to EE: failed database migrations (14.2.4) Upgrade	0	530	January 19, 2022
Background migration error Upgrade	8	771	August 11, 2023
Databasemigration fails after Update to 10.1.0 How to Use GitLab	2	1151	October 30, 2017
Upgrade to ce-10.8.3 fails on db migration Upgrade	0	693	June 3, 2018
[SOLVED] Database Migration Fails After Upgrade Upgrade	3	2976	May 10, 2017

GitLab 13.12.5 db:migrate error

Related topics