Upgarde to 14.1.8 to 16.2

How to Use GitLab Upgrade
1

HI Friends,

Trying to upgrade gitlab-ce from 14.1.8 to latest. As per the path I am trying to upgrade to 14.3.6 but migartion fails on push_event_payloads foreign-key problem. Error is as below

"rails_migration[gitlab-rails] (gitlab::database_migrations line 51) had an error: Mixlib::ShellOut::ShellCommandFailed: bash[migrate gitlab-rails database] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb line 16) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received ‘1’

---- Begin output of “bash” “/tmp/chef-script20230715-114919-cozao1” ----
STDOUT: rake aborted!
StandardError: An error has occurred, all later migrations canceled:
PG::ForeignKeyViolation: ERROR: insert or update on table “push_event_payloads” violates foreign key constraint “fk_36c74129da_tmp”

Attached is SS for ref. Please help me to check this one.

Screen Shot 2023-07-15 at 5.27.14 AM
Screen Shot 2023-07-15 at 5.27.14 AM1177×772 181 KB

Please help…

Regards
Rajesh

2

Hi, please do not post screenshots, it can virtually not be read. Instead, please copy all of that text and post it here.

Also, please use the tools that the forum provides to format your post better. Use code blocks for errors, commands and anything else that is separate from the main text, so like this:

gitlab-ctl restart

or using the error message from your first post:

"rails_migration[gitlab-rails] (gitlab::database_migrations line 51) had an error: Mixlib::ShellOut::ShellCommandFailed: bash[migrate gitlab-rails database] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb line 16) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received ‘1’

---- Begin output of “bash” “/tmp/chef-script20230715-114919-cozao1” ----
STDOUT: rake aborted!
StandardError: An error has occurred, all later migrations canceled:
PG::ForeignKeyViolation: ERROR: insert or update on table “push_event_payloads” violates foreign key constraint “fk_36c74129da_tmp”

by formatting properly, makes it easier for people to help you.

3 Likes
3

Hi,
Sorry for the inconvenience caused. Here is the error message. Please let me know if you have any solution.

rails_migration[gitlab-rails] (gitlab::database_migrations line 51) had an error: Mixlib::ShellOut::ShellCommandFailed: bash[migrate gitlab-rails database] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb line 16) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received ‘1’

---- Begin output of “bash” “/tmp/chef-script20230715-114919-cozao1” ----

STDOUT: rake aborted!

StandardError: An error has occurred, all later migrations canceled:

PG::ForeignKeyViolation: ERROR: insert or update on table “push_event_payloads” violates foreign key constraint “fk_36c74129da_tmp”

DETAIL: Key (event_id)=(23624) is not present in table “events”.

/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:293:in `block in add_concurrent_foreign_key’

/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:359:in `disable_statement_timeout’

/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:292:in `add_concurrent_foreign_key’

/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20210622045705_finalize_events_bigint_conversion.rb:40:in `swap’

/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20210622045705_finalize_events_bigint_conversion.rb:18:in `up’

/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:31:in `ddl_transaction’

/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:61:in `block (3 levels) in <top (required)>’

/opt/gitlab/embedded/bin/bundle:23:in `load’

/opt/gitlab/embedded/bin/bundle:23:in `’

4

What operating system do you have Gitlab installed on?

5

HI,

Its Ubuntu 18.04.5 LTS. I am using since 4 years.

root@gitlab_dr:/home/netadmin# cat /etc/os-release
NAME=“Ubuntu”
VERSION=“18.04.5 LTS (Bionic Beaver)”
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME=“Ubuntu 18.04.5 LTS”
VERSION_ID=“18.04”

Regards
Rajesh

6

I would first try:

gitlab-ctl reconfigure
gitlab-ctl restart
gitlab-rake db:migrate

you can also take a look at this document article: fix and retry migrations: Background migrations and upgrades | GitLab

7

HI,

Thanks for the reply. I have tried all those stuffs. am able to login gitlab console after performing above steps but getting 500 error when I click an any project. “gitlab-rake db:migrate status” shows most of the db migrations are down.

Regards
Rajesh

8

gitlab-rake db:migrate:status

down 20210830104800 Reschedule extract project topics into separate table
down 20210830140524 Add state to member
down 20210830154358 Add yaml limit constraints
down 20210831123008 Drop temporary trigger for ci sources pipelines
down 20210831134840 Add package file id channel idx to packages helm file metadata
down 20210831135249 Add installable helm pkgs idx to packages
down 20210831203408 Upsert base work item types
down 20210901044202 Push event payloads bigint conversion remove triggers
down 20210901044237 Events bigint conversion remove triggers
down 20210901065504 Add index on name and id to public groups
down 20210901153324 Slice merge request diff commit migrations
down 20210901184511 Prepare async indexes for ci builds
down 20210902144144 Drop temporary columns and triggers for ci build needs
down 20210902171406 Add latest column into the security scans table
down 20210902171808 Set default job token scope false
down 20210902184334 Add expire access tokens to doorkeeper application
down 20210903054158 Recreate stage issue events table with bigints
down 20210906100021 Delete project namespace trigger
down 20210906100316 Drop temporary columns and triggers for ci build trace chunks
down 20210907021940 Cleanup bigint conversion for ci stages
down 20210907033745 Cleanup bigint conversion for deployments
down 20210907041000 Cleanup bigint conversion for geo job artifact deleted events
down 20210907182337 Add group id fkey for user group callout
down 20210907182359 Add user id fkey for user group callout
down 20210907211557 Finalize ci builds bigint conversion
down 20210908060951 Add dast schedules to plan limits
down 20210908061132 Insert dast profile schedules plan limits
down 20210908100810 Add jobs per stage page size to application settings
down 20210908132335 Disable job token scope when unused
down 20210908140437 Add sidekiq limits to application settings
down 20210908185736 Add status to dependency proxy manifests
down 20210908185754 Add status to dependency proxy blobs
down 20210909104800 Reschedule extract project topics into separate table 2
down 20210909152027 Remove container registry enabled
down 20210909184349 Add index package id id on package files
down 20210910141043 Change ci minutes additional pack text limit
down 20210913010411 Create agent project authorizations
down 20210913010432 Add agent project authorizations foreign keys
down 20210913122457 Add namespace traversal ids to ci pending builds
down 20210914094840 Add gin index on pending builds namespace traversal ids
up 20210914095310 Cleanup orphan project access tokens
down 20210915202900 Prepare index resource group status commit id for ci builds
down 20210929144453 Add warn about potentially unwanted characters to project settings

9

I might be having a similar issue. I’m running Omnibus docker GitLab 16.1.2 and updating to 16.2. Using built in postgres 13.11.

Here’s the error output (getting the same thing when runnning rake db:migrate):

rails_migration[gitlab-rails] (gitlab::database_migrations line 51) had an error: Mixlib::ShellOut::ShellCommandFailed: bash_hide_env[migrate gitlab-rails database] (gitlab::database_migrations line 20) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of "bash"  ----
STDOUT: rake aborted!
NoMethodError: undefined method `devise' for User:Class
/opt/gitlab/embedded/service/gitlab-rails/vendor/gems/attr_encrypted/lib/attr_encrypted.rb:308:in `method_missing'
/opt/gitlab/embedded/service/gitlab-rails/vendor/gems/attr_encrypted/lib/attr_encrypted/adapters/active_record.rb:136:in `method_missing_with_attr_encrypted'
/opt/gitlab/embedded/service/gitlab-rails/app/models/user.rb:89:in `<class:User>'
/opt/gitlab/embedded/service/gitlab-rails/app/models/user.rb:5:in `<top (required)>'
<internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.0.0/rubygems/core_ext/kernel_require.rb>:37:in `require'
<internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.0.0/rubygems/core_ext/kernel_require.rb>:37:in `require'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/application_context.rb:36:in `<class:ApplicationContext>'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/application_context.rb:5:in `<module:Gitlab>'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/application_context.rb:3:in `<top (required)>'
<internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.0.0/rubygems/core_ext/kernel_require.rb>:37:in `require'
<internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.0.0/rubygems/core_ext/kernel_require.rb>:37:in `require'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/error_tracking/context_payload_generator.rb:62:in `current_context'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/error_tracking/context_payload_generator.rb:38:in `tags_payload'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/error_tracking/context_payload_generator.rb:13:in `generate'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/error_tracking/context_payload_generator.rb:7:in `generate'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/error_tracking.rb:161:in `process_exception'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/error_tracking.rb:94:in `track_and_raise_for_dev_exception'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab_settings/options.rb:154:in `method_missing'
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/1_settings.rb:33:in `<top (required)>'
/opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:7:in `<top (required)>'
<internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.0.0/rubygems/core_ext/kernel_require.rb>:37:in `require'
<internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.0.0/rubygems/core_ext/kernel_require.rb>:37:in `require'
/opt/gitlab/embedded/bin/bundle:25:in `load'
/opt/gitlab/embedded/bin/bundle:25:in `<main>'
Tasks: TOP => gitlab:db:configure => environment
(See full trace by running task with --trace)
STDERR:
---- End output of "bash"  ----
Ran "bash"  returned 1
11

When I run gitlab-ctl reconfigure, below is the error.

rake aborted!
StandardError: An error has occurred, all later migrations canceled:

PG::ForeignKeyViolation: ERROR: insert or update on table “push_event_payloads” violates foreign key constraint “fk_36c74129da_tmp”
DETAIL: Key (event_id)=(23624) is not present in table “events”.
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:293:in block in add_concurrent_foreign_key' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:359:in disable_statement_timeout’
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:292:in add_concurrent_foreign_key' /opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20210622045705_finalize_events_bigint_conversion.rb:40:in swap’
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20210622045705_finalize_events_bigint_conversion.rb:18:in up' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:31:in ddl_transaction’
/opt/gitlab/embedded/bin/bundle:23:in load' /opt/gitlab/embedded/bin/bundle:23:in ’

Caused by:
ActiveRecord::InvalidForeignKey: PG::ForeignKeyViolation: ERROR: insert or update on table “push_event_payloads” violates foreign key constraint “fk_36c74129da_tmp”
DETAIL: Key (event_id)=(23624) is not present in table “events”.
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:293:in block in add_concurrent_foreign_key' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:359:in disable_statement_timeout’
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:292:in add_concurrent_foreign_key' /opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20210622045705_finalize_events_bigint_conversion.rb:40:in swap’
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20210622045705_finalize_events_bigint_conversion.rb:18:in up' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:31:in ddl_transaction’
/opt/gitlab/embedded/bin/bundle:23:in load' /opt/gitlab/embedded/bin/bundle:23:in ’

Caused by:
PG::ForeignKeyViolation: ERROR: insert or update on table “push_event_payloads” violates foreign key constraint “fk_36c74129da_tmp”
DETAIL: Key (event_id)=(23624) is not present in table “events”.
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:293:in block in add_concurrent_foreign_key' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:359:in disable_statement_timeout’
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:292:in add_concurrent_foreign_key' /opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20210622045705_finalize_events_bigint_conversion.rb:40:in swap’
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20210622045705_finalize_events_bigint_conversion.rb:18:in up' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:31:in ddl_transaction’
/opt/gitlab/embedded/bin/bundle:23:in load' /opt/gitlab/embedded/bin/bundle:23:in ’
Tasks: TOP => db:migrate
(See full trace by running task with --trace)

12

Hi,

Anyone please help here

13

Hey has this been resolved?

1 Like
14

HI,

Not yet. could you please help ?

15

Here is an idea, try upgrading not to 14.3.6, but something lower, try any 14.2.x, or perhaps even latest 14.1.x. try upgrading to different versions, if that works then slowly move onto 14.3.6

16

HI,

Thanks for the reply. I have tried as below
14.0.12 → 14.1.0->14.1.1->14.1.8->14.2.0 …here starts some pending migrations . gitlab-rake db:migrate cleared those migrations…then did 14.2-.7>14.3.6 …Now am getting this primary key error.

Regards
Rajesh

17

Hi rajesh, you should not upgrade directly form 14 to 16, there is a sequece or also called upgrade path, we need to follow!

find the just below link

Upgrading GitLab | GitLab i hope you doing it in any of cloud provider like aws, gcp, azure and also hope you have old instance backup and you installed .deb of gitlab ce from gitlab/gitlab-ce - Installation ¡ packages.gitlab.com this! after every upgrade, you must ruin $ gitlab-ctl restart ### wait for 2 - 3 min., untill the server load ( $ htop ) avg., is under 1 and then $ sudo gitlab-rake db:migrate ### wait for 2 - 3 min., untill the server load ( $ htop ) avg., is under 1 and then $ gitlab-ctl reconfigure ### wait for 2 - 3 min., untill the server load ( $ htop ) avg., is under 1