[solved] Failed upgrade path and container restart in loop

Hi, a few days now that i try to repair my docker omnibus instance after a wrong upgrade path.

I was used to update near each monthes before, and this time i had to let my projects freezed for a long time, and machinally i start an upgrade like before : put the latest version in docker-compose and start … with a version jump it seems.

Then now the upgrade fails and the container restart in loop.
I got errors in logs (console or from volumes), and have some actions to try to resolve, but as soon as the container start, upgrade restart, fails, and then stop. Then i can’t execute commands via docker exec.

Note : i ask some advices, and was answered to try revert the dockr-file to previous version, then i’m currently trying with 14.2.3-ce.0, but think i was at a lower minor version before, i’m trying to find which in the logs ; i opened another thread to find how get my version history)

i"m trying to find how to keep the container up to work on it.

For info, here a part of reconfigure log:

---- Begin output of "bash"  "/tmp/chef-script20211022-90-9q97rx" ----
STDOUT: rake aborted!
StandardError: An error has occurred, all later migrations canceled:

Expected batched background migration for the given configuration to be marked as 'finished', but it is 'active':       {:job_class_name=>"CopyColumnUsingBackgroundMigrationJob", :table_name=>"push_event_payloads", :column_name=>"event_id", :job_arguments=>[["event_id"], ["event_id_convert_to_bigint"]]}

Finalize it manualy by running

        sudo gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,push_event_payloads,event_id,'[["event_id"]\, ["event_id_convert_to_bigint"]]']

For more information, check the documentation

        https://docs.gitlab.com/ee/user/admin_area/monitoring/background_migrations.html#database-migrations-failing-because-of-batched-background-migration-not-finished

(...)

Tasks: TOP => db:migrate
(See full trace by running task with --trace)
== 20210802043253 FinalizePushEventPayloadsBigintConversion3: migrating =======
STDERR: 
---- End output of "bash"  "/tmp/chef-script20211022-90-9q97rx" ----
Ran "bash"  "/tmp/chef-script20211022-90-9q97rx" returned 1
(END)

Pfiou finally revived !!!

For info the process done :

  • identify the last working version
    • not found by the logs (looking for this method still)
    • found with a docker image ls (luckily not cleaned) = 14.1.5
  • revert docker to 14.1.5 : OK
  • backup (data and secrets)
  • New upgrade path done :
    • 14.1.5 to 14.1.7 : OK
    • 14.1.7 to 14.2.5 : OK
    • 14.2.5 to 14.3.3 : Error at service restart , new try running : OK

image

Pfiou again !