Can't upgrade from 16.11.3 to 17.0.x - Database Migration Error

Problem to solve

Describe your question in as much detail as possible:

  • What are you seeing, and how does that differ from what you expect to see? Using the self-managed version (gitlab-ce) under Debian 11, attempting to upgrade via sudo apt update or manually attempting to update using sudo apt install gitlab-ce=17.0.0-ce.0 proceeds normally until it hits the “reconfigure” step - when it tries to run the gitlab::database_migrations step it encounters an error:
Recipe: gitlab::database_migrations
  * ruby_block[check remote PG version] action nothing (skipped due to action :nothing)
  * rails_migration[gitlab-rails] action run[2024-05-28T18:12:33-07:00] WARN: gitlab-rails does not have a log_group or default logdir mode defined. Setting to 0700.

    * bash_hide_env[migrate gitlab-rails database] action run
      [execute] rake aborted!
                StandardError: An error has occurred, all later migrations canceled:

                PG::DependentObjectsStillExist: ERROR:  cannot drop constraint ci_pipelines_pkey on table ci_pipelines because other objects depend on it
                DETAIL:  constraint fk_c262d728d4 on table p_ci_builds depends on index ci_pipelines_pkey
                HINT:  Use DROP ... CASCADE to drop the dependent objects too.
                /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/constraints_helpers.rb:284:in `drop_constraint'

I have tried multiple upgrade paths but cannot get past this migration.

Steps to reproduce

  • Start with a 16.9x or 16.11.x working installation
  • Attempt to upgrade to 17.0.1 via sudo apt upgrade (also fails the same way if manually specifying the upgrade to 17.0.0 or 17.0.1)
  • Unpacking proceeds as expected, but setup fails with an error related to an invalid constraint condition on the ci_pipelines table (Alos note that I am not using CI on this instance).
  • The apt package update is left in an incomplete state, and so I remove the 17.0.x package and re-install the 16.11.3 package.

Which troubleshooting steps have you already taken? Can you link to any docs or other resources so we know where you have been? I have tried upgrading from 16.11.3 to both 17.0.0 and 17.0.1, but both upgrade attempts fail the same way. I am ensuring that all migration jobs are complete before each upgrade step. Attempting to use sudo gitlab-ctl reconfigure also results in the same error message.

Configuration

  • Self managed gitlab-ce via apt repository
  • Debian 11 Bullseye (server, no GUI)

Versions

Please select whether options apply, and add the version information.

Versions

  • sudo gitlab-rake gitlab:env:info:
System information
System:		Debian 11
Current User:	git
Using RVM:	no
Ruby Version:	3.1.4p223
Gem Version:	3.5.7
Bundler Version:2.5.8
Rake Version:	13.0.6
Redis Version:	7.0.15
Sidekiq Version:7.1.6
Go Version:	unknown

GitLab information
Version:	16.11.3
Revision:	a52f2df1d08
Directory:	/opt/gitlab/embedded/service/gitlab-rails
DB Adapter:	PostgreSQL
DB Version:	14.11
URL:		http://sabanto:8000
HTTP Clone URL:	http://sabanto:8000/some-group/some-project.git
SSH Clone URL:	git@sabanto:some-group/some-project.git
Using LDAP:	no
Using Omniauth:	yes
Omniauth Providers: 

GitLab Shell
Version:	14.35.0
Repository storages:
- default: 	unix:/var/opt/gitlab/gitaly/gitaly.socket
GitLab Shell path:		/opt/gitlab/embedded/service/gitlab-shell

Gitaly
- default Address: 	unix:/var/opt/gitlab/gitaly/gitaly.socket
- default Version: 	16.11.3
- default Git Version: 	2.43.4

I found this topic Upgrade Gitlab version 16.11.3 → 17.0.1 which seems to cover the same issue but discusses it within the docker context - I think this issue may not be restricted to just containerization scenarios.

This issue seems relevant: Database migration error when upgrade from 16.11.2 to 17.0.0 (#462591) · Issues · GitLab.org / GitLab · GitLab

It looks like a potential fix has been merged which should be released with the 17.0.2 version. A community member also shared a workaround here.

2 Likes

@gitlab-greg - Thanks for pointing me at that post (I guess my search was too version-specific!) That looks like the problem, I’m currently sitting at 16.11.3 and I’ll just await the release of 17.0.2 to see if that takes care of things. Thanks!

2 Likes

Sounds good. 17.0.2 is set to be released on June 12.

Let us know how it goes!

1 Like