Gitlab-ce omnibus upgrade 13.2.6-ce.0 to 13.3.5-ce.0 fails on db migrate can't backout to 13.2.6-ce

caniwi · September 10, 2020, 12:40am

Tried to perform an upgrade on gitlab-ce. The db_migrate step failed with:

Recipe: gitlab::database_migrations

bash[migrate gitlab-rails database] action run
[execute] rake aborted!
StandardError: An error has occurred, all later migrations canceled:
```
      PG::InvalidObjectDefinition: ERROR:  partition "audit_events_part_5fc467ac26_201507" would overlap partition "audit_events_part_5fc467ac26_000000"
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/schema_helpers.rb:92:in `create_range_partition'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/partitioning_migration_helpers/table_management_helpers.rb:225:in `create_range_partition_safely'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/partitioning_migration_helpers/table_management_helpers.rb:207:in `create_daterange_partitions'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/partitioning_migration_helpers/table_management_helpers.rb:59:in `partition_table_by_date'
      /opt/gitlab/embedded/service/gitlab-rails/db/migrate/20200716120000_partition_audit_events.rb:13:in `up'
      /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:54:in `block (3 levels) in <top (required)>'
      /opt/gitlab/embedded/bin/bundle:23:in `load'
      /opt/gitlab/embedded/bin/bundle:23:in `<main>'
      
      Caused by:
      ActiveRecord::StatementInvalid: PG::InvalidObjectDefinition: ERROR:  partition "audit_events_part_5fc467ac26_201507" would overlap partition "audit_events_part_5fc467ac26_000000"
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/schema_helpers.rb:92:in `create_range_partition'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/partitioning_migration_helpers/table_management_helpers.rb:225:in `create_range_partition_safely'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/partitioning_migration_helpers/table_management_helpers.rb:207:in `create_daterange_partitions'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/partitioning_migration_helpers/table_management_helpers.rb:59:in `partition_table_by_date'
      /opt/gitlab/embedded/service/gitlab-rails/db/migrate/20200716120000_partition_audit_events.rb:13:in `up'
      /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:54:in `block (3 levels) in <top (required)>'
      /opt/gitlab/embedded/bin/bundle:23:in `load'
      /opt/gitlab/embedded/bin/bundle:23:in `<main>'
      
      Caused by:
      PG::InvalidObjectDefinition: ERROR:  partition "audit_events_part_5fc467ac26_201507" would overlap partition "audit_events_part_5fc467ac26_000000"
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/schema_helpers.rb:92:in `create_range_partition'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/partitioning_migration_helpers/table_management_helpers.rb:225:in `create_range_partition_safely'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/partitioning_migration_helpers/table_management_helpers.rb:207:in `create_daterange_partitions'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/partitioning_migration_helpers/table_management_helpers.rb:59:in `partition_table_by_date'
      /opt/gitlab/embedded/service/gitlab-rails/db/migrate/20200716120000_partition_audit_events.rb:13:in `up'
      /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:54:in `block (3 levels) in <top (required)>'
      /opt/gitlab/embedded/bin/bundle:23:in `load'
      /opt/gitlab/embedded/bin/bundle:23:in `<main>'
      Tasks: TOP => db:migrate
      (See full trace by running task with --trace)
      == 20200716120000 PartitionAuditEvents: migrating =============================
      -- transaction_open?()
         -> 0.0000s
      -- table_exists?("audit_events_part_5fc467ac26")
         -> 0.0008s
      -- table_exists?("gitlab_partitions_dynamic.audit_events_part_5fc467ac26_000000")
         -> 0.0007s
      -- table_exists?("gitlab_partitions_dynamic.audit_events_part_5fc467ac26_201507")
         -> 0.0006s
      -- execute("CREATE TABLE gitlab_partitions_dynamic.audit_events_part_5fc467ac26_201507 PARTITION OF audit_events_part_5fc467ac26\nFOR VALUES FROM ('2015-07-01') TO ('2015-08-01')\n")
```
================================================================================
Error executing action run on resource ‘bash[migrate gitlab-rails database]’

Mixlib::ShellOut::ShellCommandFailed

Expected process to exit with [0], but received ‘1’
---- Begin output of “bash” “/tmp/chef-script20200910-28707-1gitnef” ----
STDOUT: rake aborted!
StandardError: An error has occurred, all later migrations canceled:

PG::InvalidObjectDefinition: ERROR: partition “audit_events_part_5fc467ac26_201507” would overlap partition “audit_events_part_5fc467ac26_000000”
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/schema_helpers.rb:92:in create_range_partition' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/partitioning_migration_helpers/table_management_helpers.rb:225:in create_range_partition_safely’
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/partitioning_migration_helpers/table_management_helpers.rb:207:in create_daterange_partitions' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/partitioning_migration_helpers/table_management_helpers.rb:59:in partition_table_by_date’
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20200716120000_partition_audit_events.rb:13:in up' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:54:in block (3 levels) in <top (required)>’
/opt/gitlab/embedded/bin/bundle:23:in load' /opt/gitlab/embedded/bin/bundle:23:in ’

Caused by:
ActiveRecord::StatementInvalid: PG::InvalidObjectDefinition: ERROR: partition “audit_events_part_5fc467ac26_201507” would overlap partition “audit_events_part_5fc467ac26_000000”
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/schema_helpers.rb:92:in create_range_partition' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/partitioning_migration_helpers/table_management_helpers.rb:225:in create_range_partition_safely’
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/partitioning_migration_helpers/table_management_helpers.rb:207:in create_daterange_partitions' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/partitioning_migration_helpers/table_management_helpers.rb:59:in partition_table_by_date’
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20200716120000_partition_audit_events.rb:13:in up' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:54:in block (3 levels) in <top (required)>’
/opt/gitlab/embedded/bin/bundle:23:in load' /opt/gitlab/embedded/bin/bundle:23:in ’

Caused by:
PG::InvalidObjectDefinition: ERROR: partition “audit_events_part_5fc467ac26_201507” would overlap partition “audit_events_part_5fc467ac26_000000”
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/schema_helpers.rb:92:in create_range_partition' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/partitioning_migration_helpers/table_management_helpers.rb:225:in create_range_partition_safely’
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/partitioning_migration_helpers/table_management_helpers.rb:207:in create_daterange_partitions' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/partitioning_migration_helpers/table_management_helpers.rb:59:in partition_table_by_date’
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20200716120000_partition_audit_events.rb:13:in up' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:54:in block (3 levels) in <top (required)>’
/opt/gitlab/embedded/bin/bundle:23:in load' /opt/gitlab/embedded/bin/bundle:23:in ’
Tasks: TOP => db:migrate
(See full trace by running task with --trace)
== 20200716120000 PartitionAuditEvents: migrating =============================
– transaction_open?()
-> 0.0000s
– table_exists?(“audit_events_part_5fc467ac26”)
-> 0.0008s
– table_exists?(“gitlab_partitions_dynamic.audit_events_part_5fc467ac26_000000”)
-> 0.0007s
– table_exists?(“gitlab_partitions_dynamic.audit_events_part_5fc467ac26_201507”)
-> 0.0006s
– execute(“CREATE TABLE gitlab_partitions_dynamic.audit_events_part_5fc467ac26_201507 PARTITION OF audit_events_part_5fc467ac26\nFOR VALUES FROM (‘2015-07-01’) TO (‘2015-08-01’)\n”)
STDERR:
---- End output of “bash” “/tmp/chef-script20200910-28707-1gitnef” ----
Ran “bash” “/tmp/chef-script20200910-28707-1gitnef” returned 1

Resource Declaration:

In /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb

55: bash “migrate gitlab-rails database” do
56: code <<-EOH
57: set -e
58: log_file="#{node[‘gitlab’][‘gitlab-rails’][‘log_directory’]}/gitlab-rails-db-migrate-(date +%Y-%m-%d-%H-%M-%S).log" 59: umask 077 60: /opt/gitlab/bin/gitlab-rake gitlab:db:configure 2>& 1 | tee {log_file}
61: STATUS={PIPESTATUS[0]} 62: chown #{account_helper.gitlab_user}:#{account_helper.gitlab_group} {log_file}
63: echo $STATUS > #{db_migrate_status_file}
64: exit $STATUS
65: EOH
66: environment env_variables unless env_variables.empty?
67: notifies :run, “execute[clear the gitlab-rails cache]”, :immediately
68: dependent_services.each do |svc|
69: notifies :restart, svc, :immediately
70: end
71: not_if “(test -f #{db_migrate_status_file}) && (cat #{db_migrate_status_file} | grep -Fx 0)”
72: only_if { node[‘gitlab’][‘gitlab-rails’][‘auto_migrate’] }
73: end

Compiled Resource:

Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb:55:in `from_file’

bash(“migrate gitlab-rails database”) do
action [:run]
default_guard_interpreter :default
command nil
backup 5
interpreter “bash”
declared_type :bash
cookbook_name “gitlab”
recipe_name “database_migrations”
code " set -e\n log_file="/var/log/gitlab/gitlab-rails/gitlab-rails-db-migrate-(date +%Y-%m-%d-%H-%M-%S).log\"\n umask 077\n /opt/gitlab/bin/gitlab-rake gitlab:db:configure 2>& 1 | tee {log_file}\n STATUS={PIPESTATUS[0]}\n chown git:git {log_file}\n echo $STATUS > /var/opt/gitlab/gitlab-rails/upgrade-status/db-migrate-39b3035aa72544304a402bc15c8b1e98-467cb4161ad\n exit $STATUS\n"
domain nil
user nil
not_if “(test -f /var/opt/gitlab/gitlab-rails/upgrade-status/db-migrate-39b3035aa72544304a402bc15c8b1e98-467cb4161ad) && (cat /var/opt/gitlab/gitlab-rails/upgrade-status/db-migrate-39b3035aa72544304a402bc15c8b1e98-467cb4161ad | grep -Fx 0)”
only_if { #code block }
end

System Info:

chef_version=15.12.22
platform=ubuntu
platform_version=18.04
ruby=ruby 2.6.6p146 (2020-03-31 revision 67876) [x86_64-linux]
program_name=/opt/gitlab/embedded/bin/chef-client
executable=/opt/gitlab/embedded/bin/chef-client

Running handlers:
There was an error running gitlab-ctl reconfigure:

bash[migrate gitlab-rails database] (gitlab::database_migrations line 55) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received ‘1’
---- Begin output of “bash” “/tmp/chef-script20200910-28707-1gitnef” ----
STDOUT: rake aborted!
StandardError: An error has occurred, all later migrations canceled:

PG::InvalidObjectDefinition: ERROR: partition “audit_events_part_5fc467ac26_201507” would overlap partition “audit_events_part_5fc467ac26_000000”
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/schema_helpers.rb:92:in create_range_partition' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/partitioning_migration_helpers/table_management_helpers.rb:225:in create_range_partition_safely’
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/partitioning_migration_helpers/table_management_helpers.rb:207:in create_daterange_partitions' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/partitioning_migration_helpers/table_management_helpers.rb:59:in partition_table_by_date’
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20200716120000_partition_audit_events.rb:13:in up' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:54:in block (3 levels) in <top (required)>’
/opt/gitlab/embedded/bin/bundle:23:in load' /opt/gitlab/embedded/bin/bundle:23:in ’

Caused by:
ActiveRecord::StatementInvalid: PG::InvalidObjectDefinition: ERROR: partition “audit_events_part_5fc467ac26_201507” would overlap partition “audit_events_part_5fc467ac26_000000”
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/schema_helpers.rb:92:in create_range_partition' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/partitioning_migration_helpers/table_management_helpers.rb:225:in create_range_partition_safely’
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/partitioning_migration_helpers/table_management_helpers.rb:207:in create_daterange_partitions' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/partitioning_migration_helpers/table_management_helpers.rb:59:in partition_table_by_date’
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20200716120000_partition_audit_events.rb:13:in up' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:54:in block (3 levels) in <top (required)>’
/opt/gitlab/embedded/bin/bundle:23:in load' /opt/gitlab/embedded/bin/bundle:23:in ’

Caused by:
PG::InvalidObjectDefinition: ERROR: partition “audit_events_part_5fc467ac26_201507” would overlap partition “audit_events_part_5fc467ac26_000000”
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/schema_helpers.rb:92:in create_range_partition' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/partitioning_migration_helpers/table_management_helpers.rb:225:in create_range_partition_safely’
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/partitioning_migration_helpers/table_management_helpers.rb:207:in create_daterange_partitions' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/partitioning_migration_helpers/table_management_helpers.rb:59:in partition_table_by_date’
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20200716120000_partition_audit_events.rb:13:in up' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:54:in block (3 levels) in <top (required)>’
/opt/gitlab/embedded/bin/bundle:23:in load' /opt/gitlab/embedded/bin/bundle:23:in ’
Tasks: TOP => db:migrate
(See full trace by running task with --trace)
== 20200716120000 PartitionAuditEvents: migrating =============================
– transaction_open?()
-> 0.0000s
– table_exists?(“audit_events_part_5fc467ac26”)
-> 0.0008s
– table_exists?(“gitlab_partitions_dynamic.audit_events_part_5fc467ac26_000000”)
-> 0.0007s
– table_exists?(“gitlab_partitions_dynamic.audit_events_part_5fc467ac26_201507”)
-> 0.0006s
– execute(“CREATE TABLE gitlab_partitions_dynamic.audit_events_part_5fc467ac26_201507 PARTITION OF audit_events_part_5fc467ac26\nFOR VALUES FROM (‘2015-07-01’) TO (‘2015-08-01’)\n”)
STDERR:
---- End output of “bash” “/tmp/chef-script20200910-28707-1gitnef” ----
Ran “bash” “/tmp/chef-script20200910-28707-1gitnef” returned 1

Running handlers complete
Chef Infra Client failed. 1 resources updated in 36 seconds

And trying to restore gitlab-ce 13.2.6 I get…

apt install gitlab-ce=13.2.6-ce.0
Reading package lists… Done
Building dependency tree
Reading state information… Done
The following packages will be DOWNGRADED:
gitlab-ce
0 upgraded, 0 newly installed, 1 downgraded, 0 to remove and 18 not upgraded.
Need to get 760 MB of archives.
After this operation, 28.6 MB disk space will be freed.
Do you want to continue? [Y/n] y
Get:1 https://packages.gitlab.com/gitlab/gitlab-ce/ubuntu bionic/main amd64 gitlab-ce amd64 13.2.6-ce.0 [760 MB]
Fetched 760 MB in 1min 1s (12.4 MB/s)
dpkg: warning: downgrading gitlab-ce from 13.3.5-ce.0 to 13.2.6-ce.0
(Reading database … 236837 files and directories currently installed.)
Preparing to unpack …/gitlab-ce_13.2.6-ce.0_amd64.deb …
Malformed configuration JSON file found at /opt/gitlab/embedded/nodes/cat-porwal-prod-aklmgmt1.akl.cat-it.co.nz.json.
This usually happens when your last run of gitlab-ctl reconfigure didn’t complete successfully.
This file is used to check if any of the unsupported configurations are enabled,
and hence require a working reconfigure before upgrading.
Please run sudo gitlab-ctl reconfigure to fix it and try again.
dpkg: error processing archive /var/cache/apt/archives/gitlab-ce_13.2.6-ce.0_amd64.deb (–unpack):
new gitlab-ce package pre-installation script subprocess returned error exit status 1
Errors were encountered while processing:
/var/cache/apt/archives/gitlab-ce_13.2.6-ce.0_amd64.deb
E: Sub-process /usr/bin/dpkg returned an error code (1)

and so I am stuck… I can’t go back because the json file is messed up.

caniwi · September 10, 2020, 2:23am

Discovered for some reason the gitlab_partitions_dynamic.audit_events_part_5fc467ac26_000000 had VALUES FROM (MINVALUE) TO (‘2020-09-01 00:00:00’). Corrected that using psql, re-ran the gitlab-ctl and all was good.

Topic		Replies	Views
Upgrade to ce-10.8.3 fails on db migration Upgrade	0	693	June 3, 2018
Problem upgrade Gitlab 15.11.2-ce.0 to 15.11.9-ce.0 Upgrade issues	0	295	January 15, 2024
Omnibus upgrade failing database migration General	1	1377	December 30, 2020
Upgrade 17.2.2 -> 17.3 fails at DB migration Upgrade	4	1370	September 18, 2024
Failed upgrade to 14.2.0-ce - post installation fail on PG migrations Upgrade postgresql	4	1405	August 25, 2021

Gitlab-ce omnibus upgrade 13.2.6-ce.0 to 13.3.5-ce.0 fails on db migrate can't backout to 13.2.6-ce

================================================================================ Error executing action run on resource ‘bash[migrate gitlab-rails database]’

Mixlib::ShellOut::ShellCommandFailed

Resource Declaration:

In /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb

Compiled Resource:

Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb:55:in `from_file’

System Info:

Related topics

================================================================================
Error executing action `run` on resource ‘bash[migrate gitlab-rails database]’