Database cannot be migrated

Hello, I have finally upgrade my GitLab server to last version.

When I ran reconfigure, this error is shown:

There was an error running gitlab-ctl reconfigure:

rails_migration[gitlab-rails] (gitlab::database_migrations line 51) had an error: Mixlib::ShellOut::ShellCommandFailed: bash[migrate gitlab-rails database] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb line 16) had an error: Mixlib::ShellOut::ShellCommandFailed: Command execution failed. STDOUT/STDERR suppressed for sensitive resource

I have version 13 before upgrading. I have 14.10 now.

How can I solve that?

Thanks
Jaime

Hi @jaimestuardo

have you followed the official upgrade path 13.x.y → 13.12.15 → 14.0.12 → 14.x.y?

Hey @jaimestuardo

I ran into the same problem yesterday. For me, the instructions in this answer solved it.

Hope this helps!

Cheers
Henrik

@balonik , Yes… step by step from this page: Update & Upgrade | GitLab Administrator Guide

For instance:

13.0.14 -> 13.1.11 -> 13.8.8 -> 13.12.10 -> 13.12.12 -> 14.0.11 -> 14.1.6

Finally, yum without version to update to the latest.

Thank you, Phantom guy.
I have no idea how poking around in your suggested steps fixed my system. My errors were slightly different and much less verbose/helpful. However, just poking my system with the gitlab-rake db:migrate steps seems to have fixed my issue. You guys are lifesavers.
My issue started when I noticed that my gitlab installation wasn’t updating anymore, so I ran the following to fix a certificate issue, then all hell broke loose which lead me here:

curl -L https://packages.gitlab.com/gitlab/gitlab-ee/gpgkey | apt-key add -
apt-get update
apt-get upgrade
echo "Sorry sucka - now your system will be annoyingly broken for a week until you find this forum post"

Hello @Th3Ph4nt0m… I did what is shown in that link, and in fact, when migrate database, several errors are shown, for example,

ActiveRecord::StatementInvalid: PG::UndefinedTable: ERROR:  relation "batched_background_migration_job_transition_logs" does not exist
LINE 8:  WHERE a.attrelid = '"batched_background_migration_job_trans...

In what file should I comment some lines out? I am wondering also if it is safe to just comment a line out. Will database work without issue in the future?

Regards
Jaime

Okay, let’s go through this:

GitLab changed some relations in the database within the new update. To adopt these changes to your DB (inside your GitLab installation), you’ll need to run a migration. This migration is included in the update. However, in this migration GitLab seems to have built in a bug (or it doesn’t work for every installation). So, what you’ll need to do, is to adjust the migration file to fit your current database.

The error you’ve just posted means that the relation named batched_background_migration_job_transition_logs does not exist. So, everything that is related to this relation can’t be run.

If you are not that familiar with manually writing migrations, I advise you to post the contents of the entire migration file here, so we can tell you exactly what you need to do.

I hope this helps you!

Best
Henrik

IMPORTANT UPDATE

Update to my previous answer: I just found this post here in the forum. Maybe you should try those steps first, that could save a lot of work.

Hello @Th3Ph4nt0m , in my case, I cannot get rid of all errors.

I realized that all errors shown when migrating are the same:

Expected batched background migration for the given configuration to be marked as 'finished', but it is 'finalizing':   {:job_class_name=>"CopyColumnUsingBackgroundMigrationJob", :table_name=>"ci_builds", :column_name=>"id", :job_arguments=>[["id", "stage_id"], ["id_convert_to_bigint", "stage_id_convert_to_bigint"]]}

Finalize it manualy by running

        sudo gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,ci_builds,id,'[["id"\,"stage_id"]\,["id_convert_to_bigint"\,"stage_id_convert_to_bigint"]]']

When I ran

gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,ci_builds,id,'[["id"\,"stage_id"]\,["id_convert_to_bigint"\,"stage_id_convert_to_bigint"]]']

I got this error several times:

ActiveRecord::StatementInvalid: PG::UndefinedTable: ERROR:  relation "batched_background_migration_job_transition_logs" does not exist
LINE 8:  WHERE a.attrelid = '"batched_background_migration_job_trans...

This error is different from the post in the link, and every time I try to finalize, the same error exists.

I have found this document telling about the same error as me, but he was upgrading from 14.0.12 as in the link in your post and told that he downgraded and upgraded again.

Other post tells this:

You can apply db migration VERSION with 20211123135255 to create table batched_background_migration_job_transition_logs ，if the BatchedBackgroundMigrationJobs task performed by GitLab itself, it seems that there is no need to write this table, but manual perform is needed. so do following:

I am wondering if it is safe if I do that too.

By the way, which is the migration file?

Regards
Jaime