Forgot to wait for backgroud migrations to finish while upgrading from 14.0 to 14.4

Hi,

While upgrading from my 13.X local installation to 14.4 I forgot to wait for the migrations to finish in the 14.0 → 14.4 step, and it failed, and my gitlab is now down. Could you help me get it back up ?

Here’s the result of the migration list psql command:

gitlabhq_production=# select job_class_name, table_name, column_name, job_arguments from batched_background_migrations where status <> 3;
            job_class_name             |     table_name     | column_name |                                   job_arguments                                    
---------------------------------------+--------------------+-------------+------------------------------------------------------------------------------------
 CopyColumnUsingBackgroundMigrationJob | ci_job_artifacts   | id          | [["id", "job_id"], ["id_convert_to_bigint", "job_id_convert_to_bigint"]]
 CopyColumnUsingBackgroundMigrationJob | ci_builds          | id          | [["id", "stage_id"], ["id_convert_to_bigint", "stage_id_convert_to_bigint"]]
 CopyColumnUsingBackgroundMigrationJob | taggings           | id          | [["id", "taggable_id"], ["id_convert_to_bigint", "taggable_id_convert_to_bigint"]]
 CopyColumnUsingBackgroundMigrationJob | ci_stages          | id          | [["id"], ["id_convert_to_bigint"]]
 CopyColumnUsingBackgroundMigrationJob | ci_builds_metadata | id          | [["id"], ["id_convert_to_bigint"]]
(5 rows)

I don’t know if they are stuck or still running.

Edit: one is running, I don’t know how to see if it’s stuck.

# gitlab-rails runner -e production 'puts Gitlab::BackgroundMigration.remaining'
1

Hi,

Not sure if it will help, you can try this though:

gitlab-rake db:migrate

that might get them going again. I do remember seeing someone running specific migrations manually, there was a post on this forum about it. I’ll see if I can find it.

EDIT: could be this post: 500 Error and Internal Server Error - #11 by BenjaminLeibinger as there are examples here how to do each migrate task manually as per your list above.

Also, not sure which version it appeared in but you could go to your server for example https://gitlab.example.com/admin/background_migrations and see if it shows exactly which one. Usually you find that under Admin → Monitoring → Background Migrations

the migrate commande fails because there’s still a migration running. I tried to mark it as done without success. Now I’m using my night backup to try and go back to 13.10.0.