Unable to upgrade GitLab-EE from version 13.4.6 to 13.5.0

Hi,
We are using self-hosted GitLab docker image “gitlab/gitlab-ee:13.4.6-ee.0” and we are trying to upgrade to the version 13.5.0 but we are getting below error,

`Recipe: gitlab::database_migrations

  • bash[migrate gitlab-rails database] action run
    [execute] rake aborted!
    StandardError: An error has occurred, all later migrations canceled:

          Could not find check constraint "check_4c5e6bb0b3" on table "packages_package_files" which was referenced during the migration.
          This issue could be caused by the database schema straying from the expected state.
    
          To resolve this issue, please verify:
            1. all previous migrations have completed
            2. the database objects used in this migration match the Rails definition in schema.rb or structure.sql
    
          /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:1200:in `validate_check_constraint'
          /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:1323:in `validate_not_null_constraint'
          /opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20200915191156_validate_not_null_file_store_on_package_files.rb:13:in `up'
          /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:59:in `block (3 levels) in <top (required)>'
          /opt/gitlab/embedded/bin/bundle:23:in `load'
          /opt/gitlab/embedded/bin/bundle:23:in `<main>'
    
          Caused by:
          Could not find check constraint "check_4c5e6bb0b3" on table "packages_package_files" which was referenced during the migration.
          This issue could be caused by the database schema straying from the expected state.
    
          To resolve this issue, please verify:
            1. all previous migrations have completed
            2. the database objects used in this migration match the Rails definition in schema.rb or structure.sql
    
          /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:1200:in `validate_check_constraint'
          /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:1323:in `validate_not_null_constraint'
          /opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20200915191156_validate_not_null_file_store_on_package_files.rb:13:in `up'
          /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:59:in `block (3 levels) in <top (required)>'
          /opt/gitlab/embedded/bin/bundle:23:in `load'
          /opt/gitlab/embedded/bin/bundle:23:in `<main>'
          Tasks: TOP => db:migrate
          (See full trace by running task with --trace)
          == 20200915191156 ValidateNotNullFileStoreOnPackageFiles: migrating ===========
          -- current_schema()
             -> 0.0046s
    

    ================================================================================
    Error executing action run on resource ‘bash[migrate gitlab-rails database]’

    Mixlib::ShellOut::ShellCommandFailed

    Expected process to exit with [0], but received ‘1’
    ---- Begin output of “bash” “/tmp/chef-script20201106-30-1fjng4i” ----
    STDOUT: rake aborted!
    StandardError: An error has occurred, all later migrations canceled:

    Could not find check constraint “check_4c5e6bb0b3” on table “packages_package_files” which was referenced during the migration.
    This issue could be caused by the database schema straying from the expected state.

    To resolve this issue, please verify:

    1. all previous migrations have completed
    2. the database objects used in this migration match the Rails definition in schema.rb or structure.sql

    /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:1200:in validate_check_constraint' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:1323:in validate_not_null_constraint’
    /opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20200915191156_validate_not_null_file_store_on_package_files.rb:13:in up' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:59:in block (3 levels) in <top (required)>’
    /opt/gitlab/embedded/bin/bundle:23:in load' /opt/gitlab/embedded/bin/bundle:23:in

    Caused by:
    Could not find check constraint “check_4c5e6bb0b3” on table “packages_package_files” which was referenced during the migration.
    This issue could be caused by the database schema straying from the expected state.

    To resolve this issue, please verify:

    1. all previous migrations have completed
    2. the database objects used in this migration match the Rails definition in schema.rb or structure.sql

    /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:1200:in validate_check_constraint' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:1323:in validate_not_null_constraint’
    /opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20200915191156_validate_not_null_file_store_on_package_files.rb:13:in up' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:59:in block (3 levels) in <top (required)>’
    /opt/gitlab/embedded/bin/bundle:23:in load' /opt/gitlab/embedded/bin/bundle:23:in
    Tasks: TOP => db:migrate
    (See full trace by running task with --trace)
    == 20200915191156 ValidateNotNullFileStoreOnPackageFiles: migrating ===========
    – current_schema()
    There was an error running gitlab-ctl reconfigure:

bash[migrate gitlab-rails database] (gitlab::database_migrations line 55) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received ‘1’
---- Begin output of “bash” “/tmp/chef-script20201106-30-1fjng4i” ----
STDOUT: rake aborted!
StandardError: An error has occurred, all later migrations canceled:

Could not find check constraint “check_4c5e6bb0b3” on table “packages_package_files” which was referenced during the migration.
This issue could be caused by the database schema straying from the expected state.

To resolve this issue, please verify:

  1. all previous migrations have completed
  2. the database objects used in this migration match the Rails definition in schema.rb or structure.sql

/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:1200:in validate_check_constraint' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:1323:in validate_not_null_constraint’
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20200915191156_validate_not_null_file_store_on_package_files.rb:13:in up' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:59:in block (3 levels) in <top (required)>’
/opt/gitlab/embedded/bin/bundle:23:in load' /opt/gitlab/embedded/bin/bundle:23:in

Caused by:
Could not find check constraint “check_4c5e6bb0b3” on table “packages_package_files” which was referenced during the migration.
This issue could be caused by the database schema straying from the expected state.

To resolve this issue, please verify:

  1. all previous migrations have completed
  2. the database objects used in this migration match the Rails definition in schema.rb or structure.sql

/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:1200:in validate_check_constraint' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:1323:in validate_not_null_constraint’
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20200915191156_validate_not_null_file_store_on_package_files.rb:13:in up' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:59:in block (3 levels) in <top (required)>’
/opt/gitlab/embedded/bin/bundle:23:in load' /opt/gitlab/embedded/bin/bundle:23:in
Tasks: TOP => db:migrate
(See full trace by running task with --trace)
== 20200915191156 ValidateNotNullFileStoreOnPackageFiles: migrating ===========
– current_schema()
-> 0.0046s
STDERR:
---- End output of “bash” “/tmp/chef-script20201106-30-1fjng4i” ----
Ran “bash” “/tmp/chef-script20201106-30-1fjng4i” returned 1

   -> 0.0046s
STDERR:
---- End output of "bash"  "/tmp/chef-script20201106-30-1fjng4i" ----
Ran "bash"  "/tmp/chef-script20201106-30-1fjng4i" returned 1

Resource Declaration:
---------------------
# In /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb

 55: bash "migrate gitlab-rails database" do
 56:   code <<-EOH
 57:     set -e
 58:     log_file="#{node['gitlab']['gitlab-rails']['log_directory']}/gitlab-rails-db-migrate-$(date +%Y-%m-%d-%H-%M-%S).log"
 59:     umask 077
 60:     /opt/gitlab/bin/gitlab-rake gitlab:db:configure 2>& 1 | tee ${log_file}
 61:     STATUS=${PIPESTATUS[0]}
 62:     chown #{account_helper.gitlab_user}:#{account_helper.gitlab_group} ${log_file}
 63:     echo $STATUS > #{db_migrate_status_file}
 64:     exit $STATUS
 65:   EOH
 66:   environment env_variables unless env_variables.empty?
 67:   notifies :run, "execute[clear the gitlab-rails cache]", :immediately
 68:   dependent_services.each do |svc|
 69:     notifies :restart, svc, :immediately
 70:   end
 71:   not_if "(test -f #{db_migrate_status_file}) && (cat #{db_migrate_status_file} | grep -Fx 0)"
 72:   only_if { node['gitlab']['gitlab-rails']['auto_migrate'] }
 73: end

Compiled Resource:
------------------
# Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb:55:in `from_file'

bash("migrate gitlab-rails database") do
  action [:run]
  default_guard_interpreter :default
  command nil
  backup 5
  interpreter "bash"
  declared_type :bash
  cookbook_name "gitlab"
  recipe_name "database_migrations"
  code "    set -e\n    log_file=\"/var/log/gitlab/gitlab-rails/gitlab-rails-db-migrate-$(date +%Y-%m-%d-%H-%M-%S).log\"\n    umask 077\n    /opt/gitlab/bin/gitlab-rake gitlab:db:configure 2>& 1 | tee ${log_file}\n    STATUS=${PIPESTATUS[0]}\n    chown git:git ${log_file}\n    echo $STATUS > /var/opt/gitlab/gitlab-rails/upgrade-status/db-migrate-873248b1f0d3a7a5535771a3a1635803-b9d194b6b91\n    exit $STATUS\n"
  domain nil
  user nil
  not_if "(test -f /var/opt/gitlab/gitlab-rails/upgrade-status/db-migrate-873248b1f0d3a7a5535771a3a1635803-b9d194b6b91) && (cat /var/opt/gitlab/gitlab-rails/upgrade-status/db-migrate-873248b1f0d3a7a5535771a3a1635803-b9d194b6b91 | grep -Fx 0)"
  only_if { #code block }
end

System Info:
------------
chef_version=15.14.0
platform=ubuntu
platform_version=16.04
ruby=ruby 2.6.6p146 (2020-03-31 revision 67876) [x86_64-linux]
program_name=/opt/gitlab/embedded/bin/chef-client
executable=/opt/gitlab/embedded/bin/chef-client

Running handlers:
Running handlers complete
Chef Infra Client failed. 9 resources updated in 01 minutes 11 seconds`

Same issue here.
At the end I had 2 schemes in my database: the public and another one named just like my “dbuser”.
So, the DB:migrate end up applying the constraint in the public schema but it searchs for it in the other schema.
How did I solve it? Moving everything to the same schema (in my case, the public one).
Would be awesome if we could set the schema in the gitlab.rb conf file.