Unable to upgrade GitLab-EE from version 13.4.6 to 13.5.0

ashar_ma · November 6, 2020, 3:18pm

Hi,
We are using self-hosted GitLab docker image “gitlab/gitlab-ee:13.4.6-ee.0” and we are trying to upgrade to the version 13.5.0 but we are getting below error,

`Recipe: gitlab::database_migrations

bash[migrate gitlab-rails database] action run
[execute] rake aborted!
StandardError: An error has occurred, all later migrations canceled:
```
      Could not find check constraint "check_4c5e6bb0b3" on table "packages_package_files" which was referenced during the migration.
      This issue could be caused by the database schema straying from the expected state.

      To resolve this issue, please verify:
        1. all previous migrations have completed
        2. the database objects used in this migration match the Rails definition in schema.rb or structure.sql

      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:1200:in `validate_check_constraint'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:1323:in `validate_not_null_constraint'
      /opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20200915191156_validate_not_null_file_store_on_package_files.rb:13:in `up'
      /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:59:in `block (3 levels) in <top (required)>'
      /opt/gitlab/embedded/bin/bundle:23:in `load'
      /opt/gitlab/embedded/bin/bundle:23:in `<main>'

      Caused by:
      Could not find check constraint "check_4c5e6bb0b3" on table "packages_package_files" which was referenced during the migration.
      This issue could be caused by the database schema straying from the expected state.

      To resolve this issue, please verify:
        1. all previous migrations have completed
        2. the database objects used in this migration match the Rails definition in schema.rb or structure.sql

      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:1200:in `validate_check_constraint'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:1323:in `validate_not_null_constraint'
      /opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20200915191156_validate_not_null_file_store_on_package_files.rb:13:in `up'
      /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:59:in `block (3 levels) in <top (required)>'
      /opt/gitlab/embedded/bin/bundle:23:in `load'
      /opt/gitlab/embedded/bin/bundle:23:in `<main>'
      Tasks: TOP => db:migrate
      (See full trace by running task with --trace)
      == 20200915191156 ValidateNotNullFileStoreOnPackageFiles: migrating ===========
      -- current_schema()
         -> 0.0046s
```
================================================================================
Error executing action run on resource ‘bash[migrate gitlab-rails database]’

Mixlib::ShellOut::ShellCommandFailed

Expected process to exit with [0], but received ‘1’
---- Begin output of “bash” “/tmp/chef-script20201106-30-1fjng4i” ----
STDOUT: rake aborted!
StandardError: An error has occurred, all later migrations canceled:

Could not find check constraint “check_4c5e6bb0b3” on table “packages_package_files” which was referenced during the migration.
This issue could be caused by the database schema straying from the expected state.

To resolve this issue, please verify:
1. all previous migrations have completed
2. the database objects used in this migration match the Rails definition in schema.rb or structure.sql
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:1200:in validate_check_constraint' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:1323:in validate_not_null_constraint’
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20200915191156_validate_not_null_file_store_on_package_files.rb:13:in up' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:59:in block (3 levels) in <top (required)>’
/opt/gitlab/embedded/bin/bundle:23:in load' /opt/gitlab/embedded/bin/bundle:23:in ’

Caused by:
Could not find check constraint “check_4c5e6bb0b3” on table “packages_package_files” which was referenced during the migration.
This issue could be caused by the database schema straying from the expected state.

To resolve this issue, please verify:
1. all previous migrations have completed
2. the database objects used in this migration match the Rails definition in schema.rb or structure.sql
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:1200:in validate_check_constraint' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:1323:in validate_not_null_constraint’
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20200915191156_validate_not_null_file_store_on_package_files.rb:13:in up' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:59:in block (3 levels) in <top (required)>’
/opt/gitlab/embedded/bin/bundle:23:in load' /opt/gitlab/embedded/bin/bundle:23:in ’
Tasks: TOP => db:migrate
(See full trace by running task with --trace)
== 20200915191156 ValidateNotNullFileStoreOnPackageFiles: migrating ===========
– current_schema()
There was an error running gitlab-ctl reconfigure:

bash[migrate gitlab-rails database] (gitlab::database_migrations line 55) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received ‘1’
---- Begin output of “bash” “/tmp/chef-script20201106-30-1fjng4i” ----
STDOUT: rake aborted!
StandardError: An error has occurred, all later migrations canceled:

Could not find check constraint “check_4c5e6bb0b3” on table “packages_package_files” which was referenced during the migration.
This issue could be caused by the database schema straying from the expected state.

To resolve this issue, please verify:

all previous migrations have completed
the database objects used in this migration match the Rails definition in schema.rb or structure.sql

/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:1200:in validate_check_constraint' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:1323:in validate_not_null_constraint’
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20200915191156_validate_not_null_file_store_on_package_files.rb:13:in up' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:59:in block (3 levels) in <top (required)>’
/opt/gitlab/embedded/bin/bundle:23:in load' /opt/gitlab/embedded/bin/bundle:23:in ’

Caused by:
Could not find check constraint “check_4c5e6bb0b3” on table “packages_package_files” which was referenced during the migration.
This issue could be caused by the database schema straying from the expected state.

To resolve this issue, please verify:

all previous migrations have completed
the database objects used in this migration match the Rails definition in schema.rb or structure.sql

/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:1200:in validate_check_constraint' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:1323:in validate_not_null_constraint’
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20200915191156_validate_not_null_file_store_on_package_files.rb:13:in up' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:59:in block (3 levels) in <top (required)>’
/opt/gitlab/embedded/bin/bundle:23:in load' /opt/gitlab/embedded/bin/bundle:23:in ’
Tasks: TOP => db:migrate
(See full trace by running task with --trace)
== 20200915191156 ValidateNotNullFileStoreOnPackageFiles: migrating ===========
– current_schema()
-> 0.0046s
STDERR:
---- End output of “bash” “/tmp/chef-script20201106-30-1fjng4i” ----
Ran “bash” “/tmp/chef-script20201106-30-1fjng4i” returned 1

   -> 0.0046s
STDERR:
---- End output of "bash"  "/tmp/chef-script20201106-30-1fjng4i" ----
Ran "bash"  "/tmp/chef-script20201106-30-1fjng4i" returned 1

Resource Declaration:
---------------------
# In /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb

 55: bash "migrate gitlab-rails database" do
 56:   code <<-EOH
 57:     set -e
 58:     log_file="#{node['gitlab']['gitlab-rails']['log_directory']}/gitlab-rails-db-migrate-$(date +%Y-%m-%d-%H-%M-%S).log"
 59:     umask 077
 60:     /opt/gitlab/bin/gitlab-rake gitlab:db:configure 2>& 1 | tee ${log_file}
 61:     STATUS=${PIPESTATUS[0]}
 62:     chown #{account_helper.gitlab_user}:#{account_helper.gitlab_group} ${log_file}
 63:     echo $STATUS > #{db_migrate_status_file}
 64:     exit $STATUS
 65:   EOH
 66:   environment env_variables unless env_variables.empty?
 67:   notifies :run, "execute[clear the gitlab-rails cache]", :immediately
 68:   dependent_services.each do |svc|
 69:     notifies :restart, svc, :immediately
 70:   end
 71:   not_if "(test -f #{db_migrate_status_file}) && (cat #{db_migrate_status_file} | grep -Fx 0)"
 72:   only_if { node['gitlab']['gitlab-rails']['auto_migrate'] }
 73: end

Compiled Resource:
------------------
# Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb:55:in `from_file'

bash("migrate gitlab-rails database") do
  action [:run]
  default_guard_interpreter :default
  command nil
  backup 5
  interpreter "bash"
  declared_type :bash
  cookbook_name "gitlab"
  recipe_name "database_migrations"
  code "    set -e\n    log_file=\"/var/log/gitlab/gitlab-rails/gitlab-rails-db-migrate-$(date +%Y-%m-%d-%H-%M-%S).log\"\n    umask 077\n    /opt/gitlab/bin/gitlab-rake gitlab:db:configure 2>& 1 | tee ${log_file}\n    STATUS=${PIPESTATUS[0]}\n    chown git:git ${log_file}\n    echo $STATUS > /var/opt/gitlab/gitlab-rails/upgrade-status/db-migrate-873248b1f0d3a7a5535771a3a1635803-b9d194b6b91\n    exit $STATUS\n"
  domain nil
  user nil
  not_if "(test -f /var/opt/gitlab/gitlab-rails/upgrade-status/db-migrate-873248b1f0d3a7a5535771a3a1635803-b9d194b6b91) && (cat /var/opt/gitlab/gitlab-rails/upgrade-status/db-migrate-873248b1f0d3a7a5535771a3a1635803-b9d194b6b91 | grep -Fx 0)"
  only_if { #code block }
end

System Info:
------------
chef_version=15.14.0
platform=ubuntu
platform_version=16.04
ruby=ruby 2.6.6p146 (2020-03-31 revision 67876) [x86_64-linux]
program_name=/opt/gitlab/embedded/bin/chef-client
executable=/opt/gitlab/embedded/bin/chef-client

Running handlers:
Running handlers complete
Chef Infra Client failed. 9 resources updated in 01 minutes 11 seconds`

arthurd2 · November 26, 2020, 12:53am

Same issue here.
At the end I had 2 schemes in my database: the public and another one named just like my “dbuser”.
So, the DB:migrate end up applying the constraint in the public schema but it searchs for it in the other schema.
How did I solve it? Moving everything to the same schema (in my case, the public one).
Would be awesome if we could set the schema in the gitlab.rb conf file.

Topic		Replies	Views
Gitlab-ce - Upgrade from 15.4.6 to 15.5.0 - Unable to find check constraint on table "user_details" Upgrade community	0	592	April 18, 2023
Upgrade gitlab ee to 15.11.13 Upgrade upgrade	0	79	December 31, 2024
Fail upgrade from 14.4.0 to 14.10.5 Upgrade	1	967	July 13, 2022
Gitlab17.5.5 upgrade to 17.6.3 upgrade failure Upgrade	3	260	January 28, 2025
Update 17.8.5 to (17.9.0, 17.9.1, 17.9.2) General	2	153	March 25, 2025

Unable to upgrade GitLab-EE from version 13.4.6 to 13.5.0

================================================================================ Error executing action run on resource ‘bash[migrate gitlab-rails database]’

Mixlib::ShellOut::ShellCommandFailed

Related topics

================================================================================
Error executing action `run` on resource ‘bash[migrate gitlab-rails database]’