SOLVED: GL EE upgrade failed - cant perform ctl reconfigure

How to Use GitLab
,
1

I’ve upgraded gitlab-ee with apt and now gitlab-ctl reconfigure fails

  • upgraded gitlab-ee:amd64 from 17.1.2-ee.to 17.2.0-ee
  • executed gitlab-ctl upgrade without any problems
  • executed gitlab-ctl reconfigure but failed
  • gitlab runs seemingly normal

Steps to reproduce

  • gitlab-ctl reconfigure will fail everytime

Logfiles

[...]
StandardError: An error has occurred, this and all later migrations canceled:
PG::DependentObjectsStillExist: ERROR:  cannot drop desired object(s) because other objects depend on them
DETAIL:  default value for column id of table vulnerability_user_mentions depends on sequence vulnerability_user_mentions_id_seq
[...]

gitlab-rake db:migrate:status: https://pastebin.com/raw/1dCzsGF2

[...]
   up     20240703104831  post     17.3         AddFkFromPCiBuildTagsToTags
  down    20240709014310  post     17.3         CleanupBigintConversionsForPCiBuildsAttempt2
  down    20240709060152  post     17.3         FinalizeBackfillFindingInitialPipelineId
  down    20240711092719  post     17.3         EnsureUniqueIdForPCiJobAnnotation

stacktrace “/opt/gitlab/embedded/cookbooks/cache/cinc-stacktrace.out”: https://pastebin.com/raw/RhKHPK01

From this topic I’ve already took advice from gitlab-greg and checked if there are any pending / batched / failed background jobs but all seems normal

Configuration

  • no performed changes except gitlab-ctl upgrade / reconfigure

Versions

3 Likes
2

I have the same issue with GitLab CE omnibus on Debian, after updating it with aptitude.

PG::DependentObjectsStillExist: ERROR:  cannot drop desired object(s) because other objects depend on them
      DETAIL:  default value for column id of table merge_request_assignees depends on sequence merge_request_assignees_id_seq
      HINT:  Use DROP ... CASCADE to drop the dependent objects too.

The system is now bricked and I do not know how to roll back, and if it is safe.

Rolling back is in general not a good idea because we need to update every software sooner or later.

How can this problem be solved?

gitlab-rake db:migrate:status: also shows exactly the same

   up     20240703104831  post     17.3         AddFkFromPCiBuildTagsToTags
  down    20240709014310  post     17.3         CleanupBigintConversionsForPCiBuildsAttempt2
  down    20240709060152  post     17.3         FinalizeBackfillFindingInitialPipelineId
  down    20240711092719  post     17.3         EnsureUniqueIdForPCiJobAnnotation

EDIT: I created a ticket GitLab CE omnibus bricked after upgrade (PG::DependentObjectsStillExist) (#473406) · Issues · GitLab.org / GitLab · GitLab

1 Like
3

Same issue here, unable to run gitlab-ctl reconfigure, encountering the same error with sequence management.

1 Like
4

Exact same issue here. Hope GitLab will provide a hotfix via the omnibus package :confused:

5

The issues discuss queries for analyzing the wrong ownership permissions.

  1. Query in Upgrade Failed 17.1.2 -> 17.2.0 at database_migrations in Docker (#473337) · Issues · GitLab.org / GitLab · GitLab and user response in the thread on a fix query.
  2. User example and fix in Upgrade Failed 17.1.2 -> 17.2.0 at database_migrations in Docker (#473337) · Issues · GitLab.org / GitLab · GitLab
  3. User example in CleanupBigintConversionsForPCiBuilds fails due to commit_id_convert_to_bigint owning sequences (#468541) · Issues · GitLab.org / GitLab · GitLab

There’s also a “catch all” SQL query file in Upgrade Failed 17.1.2 -> 17.2.0 at database_migrations in Docker (#473337) · Issues · GitLab.org / GitLab · GitLab

I’d suggest continuing in the issue discussions and report feedback on the suggested queries.

6

I fixed the error with:

  1. Identify the error statement:
    gitlab-psql:
    SELECT seq_pg_class.relname AS seq_name, dep_pg_class.relname AS table_name, pg_attribute.attname AS col_name FROM pg_class seq_pg_class INNER JOIN pg_depend ON seq_pg_class.oid = pg_depend.objid INNER JOIN pg_class dep_pg_class ON pg_depend.refobjid = dep_pg_class.oid INNER JOIN pg_attribute ON dep_pg_class.oid = pg_attribute.attrelid AND pg_depend.refobjsubid = pg_attribute.attnum WHERE seq_pg_class.relkind = ‘S’ AND (dep_pg_class.relname = ‘ci_builds’ OR dep_pg_class.relname = ‘p_ci_builds’);

It showed me something with:

 seq_name     | table_name  | col_name

------------------±------------±---------
ci_builds_id_seq | p_ci_builds | id
vulnerability_user_mentions.id …

  1. Fix the error (Found in Upgrade Failed 17.1.2 -> 17.2.0 at database_migrations in Docker (#473337) · Issues · GitLab.org / GitLab · GitLab from Stan Hu · GitLab):
    ALTER SEQUENCE public.vulnerability_user_mentions_id_seq OWNED BY vulnerability_user_mentions.id;

  2. gitlab-ctl reconfigure worked
    Problem is solved

1 Like