Gitlab Upgrade Fails from 16.4.1 to 16.6.0 (latest version)

General
1

We have recently tried to upgrade Gitlab from version 16.4.1 to 16.6.0

The upgrade fails due to following reasons:

Caused by:

PG::WrongObjectType: ERROR: “p_ci_builds” is a partitioned table
DETAIL: Partitioned tables cannot have BEFORE / FOR EACH ROW triggers.
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/unidirectional_copy_trigger.rb:40:in create' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:531:in install_rename_triggers’
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:1182:in block in create_temporary_columns_and_triggers' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:123:in run_block’
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:134:in block in run_block_with_lock_timeout' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:129:in run_block_with_lock_timeout’
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:97:in run' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retries_helpers.rb:52:in with_lock_retries’
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/v2.rb:99:in with_lock_retries' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:1151:in create_temporary_columns_and_triggers’
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:735:in initialize_conversion_of_integer_to_bigint' /opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20231031134320_init_conversion_for_p_ci_builds_for_self_host.rb:25:in up’
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:33:in block in exec_migration' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/query_analyzer.rb:37:in within’
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:30:in exec_migration' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/automatic_lock_writes_on_tables.rb:21:in exec_migration’
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:36:in ddl_transaction' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/runner_backoff/active_record_mixin.rb:21:in execute_migration_in_transaction’
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:15:in block in with_advisory_lock_connection' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:12:in with_advisory_lock_connection’
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:126:in configure_database' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:101:in block (3 levels) in <top (required)>’
/opt/gitlab/embedded/bin/bundle:25:in load' /opt/gitlab/embedded/bin/bundle:25:in ’
Tasks: TOP => db:migrate
(See full trace by running task with --trace)
STDERR:
---- End output of “bash” ----
Ran “bash” returned 1

Some details are:

  1. OS Ubuntu 20.04
  2. Gitlab PG is 13.11

very awkward thing is that we when we tried to start using gitlab-ctl all services started working and gitlab is even showing that it is upgraded to latest version, however the gitlab-ctl reconfigure is still failing.

Does anyone have faced the same issue? And what solution did you applied?

2

@gitlab-matthew @gitlab-greg @gitlabSonam pls assist!

3

We came to know following:

down 20231031134320 Init conversion for p ci builds for self host
down 20231031141439 Add smtp authentication to service desk custom email credentials
down 20231031200433 Add framework fk to compliance framework security policies
down 20231031200645 Add policy configuration fk to compliance framework security policies
down 20231101130230 Remove in product marketing emails campaign column
down 20231102083539 Backfill p ci builds pipeline
down 20231102142553 Add zoekt nodes
down 20231102142554 Migrate zoekt shards to zoekt nodes
down 20231102142555 Add zoekt node id to indexed namespaces
down 20231102142557 Remove zoekt shard null constraint from indexed namespaces
down 20231102142565 Add zoekt node foreign key to indexed namespaces
down 20231103132849 Add state index for snippet repository storage move
down 20231103162825 Add wolfi purl type to package metadata purl types
down 20231103195309 Remove deprecated package metadata sync worker
down 20231103223224 Backfill zoekt node id on indexed namespaces
down 20231105165706 Drop repositories columns from geo node status table
down 20231106145853 Add product analytics enabled to namespace settings
down 20231106212340 Add visibility level to catalog resources
down 20231107062104 Add network policy egress to agent
down 20231107071201 Add project authorizations recalculated at to user details
down 20231107205734 Add update namespace name to application settings
down 20231108072342 Add display time format preference
down 20231108093031 Add allow project creation for guest and below to application settings
down 20231109133153 Drop idx namespaces on ldap sync last successful update at for gitlab
down 20231109183438 Drop merge request assignees on merge request id index

How to do DB migrations?

4

Wrangling Database Migrations can be tricky as there are multiple types of migration, but based on the output of sudo gitlab-rake db:migrate:status you shared it looks like these are all recent DB migrations (as indicated by YYYYMMDD[…] prefix).

I suggest following these instructions to see if we can get all the migrations in an “up” state: Maintenance Rake tasks | GitLab

Good luck! :crossed_fingers:

1 Like
5

Hi @gitlab-greg

We tried executing “gitlab-rake db:migrate:up 20231031134320 --trace”

Output is:

** Invoke db:migrate:up (first_time)
** Invoke db:load_config (first_time)
** Invoke environment (first_time)
** Execute environment

      ██     ██  █████  ██████  ███    ██ ██ ███    ██  ██████ 
      ██     ██ ██   ██ ██   ██ ████   ██ ██ ████   ██ ██      
      ██  █  ██ ███████ ██████  ██ ██  ██ ██ ██ ██  ██ ██   ███ 
      ██ ███ ██ ██   ██ ██   ██ ██  ██ ██ ██ ██  ██ ██ ██    ██ 
       ███ ███  ██   ██ ██   ██ ██   ████ ██ ██   ████  ██████

You are using PostgreSQL 12.7 for the main database, but this version of GitLab requires PostgreSQL >= 13.

Please upgrade your environment to a supported PostgreSQL version. See
Installation system requirements | GitLab for details.

      ██     ██  █████  ██████  ███    ██ ██ ███    ██  ██████ 
      ██     ██ ██   ██ ██   ██ ████   ██ ██ ████   ██ ██      
      ██  █  ██ ███████ ██████  ██ ██  ██ ██ ██ ██  ██ ██   ███ 
      ██ ███ ██ ██   ██ ██   ██ ██  ██ ██ ██ ██  ██ ██ ██    ██ 
       ███ ███  ██   ██ ██   ██ ██   ████ ██ ██   ████  ██████

You are using PostgreSQL 12.7 for the ci database, but this version of GitLab requires PostgreSQL >= 13.

Please upgrade your environment to a supported PostgreSQL version. See
Installation system requirements | GitLab for details.

** Execute db:load_config
** Execute db:migrate:up
rake aborted!
VERSION is required
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/activerecord-7.0.8/lib/active_record/railties/databases.rake:191:in block (3 levels) in <top (required)>' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/rake-13.0.6/lib/rake/task.rb:281:in block in execute’
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/rake-13.0.6/lib/rake/task.rb:281:in each' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/rake-13.0.6/lib/rake/task.rb:281:in execute’
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/sentry-ruby-5.8.0/lib/sentry/rake.rb:26:in execute' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/rake-13.0.6/lib/rake/task.rb:219:in block in invoke_with_call_chain’
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/rake-13.0.6/lib/rake/task.rb:199:in synchronize' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/rake-13.0.6/lib/rake/task.rb:199:in invoke_with_call_chain’
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/rake-13.0.6/lib/rake/task.rb:188:in invoke' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/rake-13.0.6/lib/rake/application.rb:160:in invoke_task’
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/rake-13.0.6/lib/rake/application.rb:116:in block (2 levels) in top_level' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/rake-13.0.6/lib/rake/application.rb:116:in each’
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/rake-13.0.6/lib/rake/application.rb:116:in block in top_level' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/rake-13.0.6/lib/rake/application.rb:125:in run_with_threads’
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/rake-13.0.6/lib/rake/application.rb:110:in top_level' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/rake-13.0.6/lib/rake/application.rb:83:in block in run’
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/rake-13.0.6/lib/rake/application.rb:186:in standard_exception_handling' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/rake-13.0.6/lib/rake/application.rb:80:in run’
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/rake-13.0.6/exe/rake:27:in <top (required)>' /opt/gitlab/embedded/bin/rake:25:in load’
/opt/gitlab/embedded/bin/rake:25:in <top (required)>' /opt/gitlab/embedded/lib/ruby/site_ruby/3.0.0/bundler/cli/exec.rb:58:in load’
/opt/gitlab/embedded/lib/ruby/site_ruby/3.0.0/bundler/cli/exec.rb:58:in kernel_load' /opt/gitlab/embedded/lib/ruby/site_ruby/3.0.0/bundler/cli/exec.rb:23:in run’
/opt/gitlab/embedded/lib/ruby/site_ruby/3.0.0/bundler/cli.rb:492:in exec' /opt/gitlab/embedded/lib/ruby/site_ruby/3.0.0/bundler/vendor/thor/lib/thor/command.rb:27:in run’
/opt/gitlab/embedded/lib/ruby/site_ruby/3.0.0/bundler/vendor/thor/lib/thor/invocation.rb:127:in invoke_command' /opt/gitlab/embedded/lib/ruby/site_ruby/3.0.0/bundler/vendor/thor/lib/thor.rb:392:in dispatch’
/opt/gitlab/embedded/lib/ruby/site_ruby/3.0.0/bundler/cli.rb:34:in dispatch' /opt/gitlab/embedded/lib/ruby/site_ruby/3.0.0/bundler/vendor/thor/lib/thor/base.rb:485:in start’
/opt/gitlab/embedded/lib/ruby/site_ruby/3.0.0/bundler/cli.rb:28:in start' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/bundler-2.4.21/exe/bundle:37:in block in <top (required)>’
/opt/gitlab/embedded/lib/ruby/site_ruby/3.0.0/bundler/friendly_errors.rb:117:in with_friendly_errors' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/bundler-2.4.21/exe/bundle:29:in <top (required)>’
/opt/gitlab/embedded/bin/bundle:25:in load' /opt/gitlab/embedded/bin/bundle:25:in ’
Tasks: TOP => db:migrate:up

the binaries of Postgresql in 13.11 but when we execute “gitlab-rake gitlab:env:info” it say the database version is

GitLab information
Version: 16.6.0-ee
Revision: 94991886af3
Directory: /opt/gitlab/embedded/service/gitlab-rails
DB Adapter: PostgreSQL
DB Version: 12.7

6

It’s odd that gitlab-rake gitlab:env:info shows 12.7 since PostgreSQL 12.7 hasn’t been default since GitLab 15.x PostgreSQL versions shipped with the Linux package | GitLab :thinking:

Are you using the GitLab Omnibus bundled PostgreSQL on a single node, or an external PostgreSQL database?

In any case, the error you’re seeing suggests like the PostgreSQL upgrade did not successfully complete.

You can try to do this manually with Database settings | GitLab

2 Likes
7

@gitlab-greg Hi Greg
Thanks for you support, we found the problem, the database was setup externally and there was also a database internally same on the server so gitlabctl was point to local DB and external database was version 12, we moved the gitlab database to a new server with version 14 and migrated all the data as well, after migration we did gitlab-ctl reconfigure and the so far all is working fine!

Thank you so much for you assistance, during this troubleshooting we came to know alot about gitlab!

1 Like
8

Glad to hear you figured it out and got it working @mbanjum ! Thanks for sharing what you did to fix it.

1 Like