Failed upgrade to 14.2.0-ce - post installation fail on PG migrations

How to Use GitLab Upgrade
1

The upgrade process runs smoothly, it fails on the post inst reconfigure at this point:

Running handlers:
There was an error running gitlab-ctl reconfigure:

rails_migration[gitlab-rails] (gitlab::database_migrations line 51) had an error: Mixlib::ShellOut::ShellCommandFailed: bash[migrate gitlab-rails database] (/opt/gitlab/embedded/co
okbooks/cache/cookbooks/gitlab/resources/rails_migration.rb line 16) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received ‘1’
---- Begin output of “bash” “/tmp/chef-script20210823-24732-hy4m23” ----
STDOUT: rake aborted!
StandardError: An error has occurred, all later migrations canceled:

PG::InvalidObjectDefinition: ERROR: functions in index predicate must be marked IMMUTABLE
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:154:in block in add_concurrent_index' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:358:in disable_statement_timeout’
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:153:in add_concurrent_index' /opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20210809123658_orphaned_invite_tokens_cleanup.rb:14:in up’
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:61:in block (3 levels) in <top (required)>' /opt/gitlab/embedded/bin/bundle:23:in load’
/opt/gitlab/embedded/bin/bundle:23:in `’

Caused by:
ActiveRecord::StatementInvalid: PG::InvalidObjectDefinition: ERROR: functions in index predicate must be marked IMMUTABLE
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:154:in block in add_concurrent_index' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:358:in disable_statement_timeout’
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:153:in add_concurrent_index' /opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20210809123658_orphaned_invite_tokens_cleanup.rb:14:in up’
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:61:in block (3 levels) in <top (required)>' /opt/gitlab/embedded/bin/bundle:23:in load’
/opt/gitlab/embedded/bin/bundle:23:in `’

Caused by:
PG::InvalidObjectDefinition: ERROR: functions in index predicate must be marked IMMUTABLE
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:154:in block in add_concurrent_index' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:358:in disable_statement_timeout’
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:153:in add_concurrent_index' /opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20210809123658_orphaned_invite_tokens_cleanup.rb:14:in up’
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:61:in block (3 levels) in <top (required)>' /opt/gitlab/embedded/bin/bundle:23:in load’
/opt/gitlab/embedded/bin/bundle:23:in `’
Tasks: TOP => db:migrate
(See full trace by running task with --trace)
== 20210809123658 OrphanedInviteTokensCleanup: migrating ======================
– transaction_open?()
→ 0.0000s
– index_exists?(“members”, :id, {:where=>“invite_token IS NOT NULL and invite_accepted_at IS NOT NULL and invite_accepted_at < created_at”, :name=>“tmp_idx_orphaned_invite_tokens”, :algorithm=>:concurrently})
→ 0.0163s
– execute(“SET statement_timeout TO 0”)
→ 0.0006s
– add_index(“members”, :id, {:where=>“invite_token IS NOT NULL and invite_accepted_at IS NOT NULL and invite_accepted_at < created_at”, :name=>“tmp_idx_orphaned_invite_tokens”, :algorithm=>:concurrently})
– execute(“RESET statement_timeout”)
→ 0.0004s
STDERR:
---- End output of “bash” “/tmp/chef-script20210823-2468-1jwvfsv” ----
Ran “bash” “/tmp/chef-script20210823-2468-1jwvfsv” returned 1

Running handlers complete
Chef Infra Client failed. 0 resources updated in 42 seconds

There was an error running gitlab-ctl reconfigure. Please check the output above for more
details.

2

We have the same issue upgrading from gitlab-14.1.2-ee to 14.2.1-ee

3

I’ve performed the update using the command. sudo gitlab-rake gitlab:db:mark_migration_complete[20210809123658]. Not sure if is just a temporary workaround or there will be repercussions.

4

We hit the same issue when trying to upgrade from 14.1.3 to 14.2.1.

Aug 24 09:57:27 /home/git/gitlab/lib/gitlab/database/migration_helpers.rb:154:in `block in add_concurrent_index'
Aug 24 09:57:27 /home/git/gitlab/lib/gitlab/database/migration_helpers.rb:353:in `disable_statement_timeout'
Aug 24 09:57:27 /home/git/gitlab/lib/gitlab/database/migration_helpers.rb:153:in `add_concurrent_index'
Aug 24 09:57:27 /home/git/gitlab/db/post_migrate/20210809123658_orphaned_invite_tokens_cleanup.rb:14:in `up'
Aug 24 09:57:27 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/migration.rb:870:in `public_send'
Aug 24 09:57:27 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/migration.rb:870:in `exec_migration'
Aug 24 09:57:27 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/migration.rb:851:in `block (2 levels) in migrate'
Aug 24 09:57:27 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/migration.rb:850:in `block in migrate'
Aug 24 09:57:27 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:462:in `with_connection'
Aug 24 09:57:27 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/migration.rb:849:in `migrate'
Aug 24 09:57:27 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/migration.rb:1037:in `migrate'
Aug 24 09:57:27 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/migration.rb:1329:in `block in execute_migration_in_transaction'
Aug 24 09:57:27 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/migration.rb:1382:in `ddl_transaction'
Aug 24 09:57:27 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/migration.rb:1328:in `execute_migration_in_transaction'
Aug 24 09:57:27 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/migration.rb:1302:in `each'
Aug 24 09:57:27 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/migration.rb:1302:in `migrate_without_lock'
Aug 24 09:57:27 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/migration.rb:1251:in `block in migrate'
Aug 24 09:57:27 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/migration.rb:1401:in `block in with_advisory_lock'
Aug 24 09:57:27 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/migration.rb:1416:in `block in with_advisory_lock_connection'
Aug 24 09:57:27 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:462:in `with_connection'
Aug 24 09:57:27 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/migration.rb:1416:in `with_advisory_lock_connection'
Aug 24 09:57:27 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/migration.rb:1397:in `with_advisory_lock'
Aug 24 09:57:27 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/migration.rb:1251:in `migrate'
Aug 24 09:57:27 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/migration.rb:1086:in `up'
Aug 24 09:57:27 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/migration.rb:1061:in `migrate'
Aug 24 09:57:27 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/tasks/database_tasks.rb:237:in `migrate'
Aug 24 09:57:27 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/railties/databases.rake:92:in `block (3 levels) in <top (required)>'
Aug 24 09:57:27 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/railties/databases.rake:90:in `each'
Aug 24 09:57:27 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/railties/databases.rake:90:in `block (2 levels) in <top (required)>'
Aug 24 09:57:27 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/rake-13.0.6/exe/rake:27:in `<top (required)>'
Aug 24 09:57:27
Aug 24 09:57:27 Caused by:
Aug 24 09:57:27 PG::InvalidObjectDefinition: ERROR: functions in index predicate must be marked IMMUTABLE
Aug 24 09:57:27 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:47:in `exec'
Aug 24 09:57:27 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:47:in `block (2 levels) in execute'
Aug 24 09:57:27 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.3.2/lib/active_support/dependencies/interlock.rb:48:in `block in permit_concurrent_loads'
Aug 24 09:57:27 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.3.2/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares'
Aug 24 09:57:27 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.3.2/lib/active_support/dependencies/interlock.rb:47:in `permit_concurrent_loads'
Aug 24 09:57:27 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:46:in `block in execute'
Aug 24 09:57:27 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/connection_adapters/abstract_adapter.rb:696:in `block (2 levels) in log'
Aug 24 09:57:27 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.3.2/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in `block (2 levels) in synchronize'
Aug 24 09:57:27 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.3.2/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
Aug 24 09:57:27 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.3.2/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
Aug 24 09:57:27 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.3.2/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
Aug 24 09:57:27 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.3.2/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
Aug 24 09:57:27 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/connection_adapters/abstract_adapter.rb:695:in `block in log'
Aug 24 09:57:27 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.3.2/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
Aug 24 09:57:27 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/connection_adapters/abstract_adapter.rb:687:in `log'
Aug 24 09:57:27 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:45:in `execute'
Aug 24 09:57:27 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/marginalia-1.10.0/lib/marginalia.rb:71:in `execute_with_marginalia'
Aug 24 09:57:27 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/connection_adapters/postgresql/schema_statements.rb:448:in `add_index'
Aug 24 09:57:27 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/migration.rb:929:in `block in method_missing'
Aug 24 09:57:27 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/migration.rb:897:in `block in say_with_time'
Aug 24 09:57:27 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/migration.rb:897:in `say_with_time'
Aug 24 09:57:27 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/migration.rb:918:in `method_missing'
Aug 24 09:57:27 /home/git/gitlab/lib/gitlab/database/migration_helpers.rb:154:in `block in add_concurrent_index'
Aug 24 09:57:27 /home/git/gitlab/lib/gitlab/database/migration_helpers.rb:353:in `disable_statement_timeout'
Aug 24 09:57:27 /home/git/gitlab/lib/gitlab/database/migration_helpers.rb:153:in `add_concurrent_index'
Aug 24 09:57:27 /home/git/gitlab/db/post_migrate/20210809123658_orphaned_invite_tokens_cleanup.rb:14:in `up'
Aug 24 09:57:27 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/migration.rb:870:in `public_send'
Aug 24 09:57:27 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/migration.rb:870:in `exec_migration'
Aug 24 09:57:27 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/migration.rb:851:in `block (2 levels) in migrate'
Aug 24 09:57:27 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/migration.rb:850:in `block in migrate'
Aug 24 09:57:27 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:462:in `with_connection'
Aug 24 09:57:27 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/migration.rb:849:in `migrate'
Aug 24 09:57:27 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/migration.rb:1037:in `migrate'
Aug 24 09:57:27 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/migration.rb:1329:in `block in execute_migration_in_transaction'
Aug 24 09:57:27 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/migration.rb:1382:in `ddl_transaction'
Aug 24 09:57:27 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/migration.rb:1328:in `execute_migration_in_transaction'
Aug 24 09:57:27 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/migration.rb:1302:in `each'
Aug 24 09:57:27 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/migration.rb:1302:in `migrate_without_lock'
Aug 24 09:57:27 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/migration.rb:1251:in `block in migrate'
Aug 24 09:57:27 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/migration.rb:1401:in `block in with_advisory_lock'
Aug 24 09:57:27 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/migration.rb:1416:in `block in with_advisory_lock_connection'
Aug 24 09:57:27 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:462:in `with_connection'
Aug 24 09:57:27 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/migration.rb:1416:in `with_advisory_lock_connection'
Aug 24 09:57:27 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/migration.rb:1397:in `with_advisory_lock'
Aug 24 09:57:27 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/migration.rb:1251:in `migrate'
Aug 24 09:57:27 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/migration.rb:1086:in `up'
Aug 24 09:57:27 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/migration.rb:1061:in `migrate'
Aug 24 09:57:27 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/tasks/database_tasks.rb:237:in `migrate'
Aug 24 09:57:27 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/railties/databases.rake:92:in `block (3 levels) in <top (required)>'
Aug 24 09:57:27 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/railties/databases.rake:90:in `each'
Aug 24 09:57:27 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/railties/databases.rake:90:in `block (2 levels) in <top (required)>'
Aug 24 09:57:27 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/rake-13.0.6/exe/rake:27:in `<top (required)>'
Aug 24 09:57:27 Tasks: TOP => db:migrate
5

that did seem a little extreme to me, this is what I did:

edited /opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20210809123658_orphaned_invite_tokens_cleanup.rb

and commented out line 14

#add_concurrent_index('members', :id, where: QUERY_CONDITION, name: TMP_INDEX_NAME)

counting on the fact that the index SEEMS to be created and dropped only to be used during the migration.