Migration path is not working

alansenairj · March 2, 2023, 9:02pm

It is very hard to follow upgrade path :
12.0.12 => 12.1.17 => 12.10.14 => 13.0.14 => 13.1.11 => 13.8.8 => 13.12.15 => 14.0.12 => 14.3.6 => 14.9.5 => 14.10.5 => 15.0.5 => 15.4.6 => 15.9.1

When it gets to gitlab-ce-13.12.15 version it shows to me a message to migrate to hash.
gitlab-rake gitlab:storage:migrate_to_hashed
Then I command touch /etc/gitlab/skip-unmigrated-data-check

yum install -y gitlab-ce-14.0.12
gitlab-ctl reconfigure
gitlab-ctl restart

When I install yum install -y gitlab-ce-14.3.6 it crashes.
"
Running transaction
Malformed configuration JSON file found at /opt/gitlab/embedded/nodes/gitlab3.csivix.com.br.json.
This usually happens when your last run of gitlab-ctl reconfigure didn’t complete successfully.
This file is used to check if any of the unsupported configurations are enabled,
and hence require a working reconfigure before upgrading.
Please run sudo gitlab-ctl reconfigure to fix it and try again.
error: %pre(gitlab-ce-14.10.5-ce.0.el7.x86_64) scriptlet failed, exit status 1
Error in PREIN scriptlet in rpm package gitlab-ce-14.10.5-ce.0.el7.x86_64
gitlab-ce-14.9.5-ce.0.el7.x86_64 was supposed to be removed but is not!
Verifying : gitlab-ce-14.9.5-ce.0.el7.x86_64 1/2
Verifying : gitlab-ce-14.10.5-ce.0.el7.x86_64 2/2

Failed:
gitlab-ce.x86_64 0:14.9.5-ce.0.el7 gitlab-ce.x86_64 0:14.10.5-ce.0.el7
"

I issued gitlab-ctl reconfigure and I got error

  ================================================================================
  Error executing action `run` on resource 'bash[migrate gitlab-rails database]'
  ================================================================================
  
  Mixlib::ShellOut::ShellCommandFailed
  ------------------------------------
  Expected process to exit with [0], but received '1'
  ---- Begin output of "bash"  "/tmp/chef-script20230302-12514-17hj9jw" ----
  STDOUT: rake aborted!
  StandardError: An error has occurred, all later migrations canceled:
  
  Expected batched background migration for the given configuration to be marked as 'finished', but it is 'active':	{:job_class_name=>"CopyColumnUsingBackgroundMigrationJob", :table_name=>"ci_stages", :column_name=>"id", :job_arguments=>[["id"], ["id_convert_to_bigint"]]}

Caused by:
Expected batched background migration for the given configuration to be marked as ‘finished’, but it is ‘active’: {:job_class_name=>“CopyColumnUsingBackgroundMigrationJob”, :table_name=>“ci_stages”, :column_name=>“id”, :job_arguments=>[[“id”], [“id_convert_to_bigint”]]}

I cannot end my upgrade. I nedd some help to go on it.

gitlab-greg · March 2, 2023, 9:43pm

Hi @alansenairj

When I install yum install -y gitlab-ce-14.3.6 it crashes.

Based on the error message you’ve pasted, it appears that you’re trying to upgrade from 14.9.5 to 14.10.5. Is it possible that you’ve skipped 14.3.6?

error: %pre(gitlab-ce-14.10.5-ce.0.el7.x86_64) scriptlet failed, exit status 1
Error in PREIN scriptlet in rpm package gitlab-ce-14.10.5-ce.0.el7.x86_64
gitlab-ce-14.9.5-ce.0.el7.x86_64 was supposed to be removed but is not!
Verifying : gitlab-ce-14.9.5-ce.0.el7.x86_64 1/2
Verifying : gitlab-ce-14.10.5-ce.0.el7.x86_64 2/2

Failed:
gitlab-ce.x86_64 0:14.9.5-ce.0.el7 gitlab-ce.x86_64 0:14.10.5-ce.0.el7

The rest of the error message:

 Mixlib::ShellOut::ShellCommandFailed
  ------------------------------------
  Expected process to exit with [0], but received '1'
  ---- Begin output of "bash"  "/tmp/chef-script20230302-12514-17hj9jw" ----
  STDOUT: rake aborted!
  StandardError: An error has occurred, all later migrations canceled:
  
  Expected batched background migration for the given configuration to be marked as 'finished', but it is 'active':	{:job_class_name=>"CopyColumnUsingBackgroundMigrationJob", :table_name=>"ci_stages", :column_name=>"id", :job_arguments=>[["id"], ["id_convert_to_bigint"]]}

Indicates a mandatory background migration did not complete - that’s what’s blocking your upgrade.

To fix this, refer to the following resources:

alansenairj · March 3, 2023, 1:09pm

Hi Greg. I clear all things and restarted the migration process. Let me explain it better.

Firt my gitlab server is using 11.8.2ee version. I bring my server backup to another VM and restore back there.
I upgraded it to 11.11.8 ee and then I use that site to help me with upgrade path. It gave me this path:

12.0.12 => 12.1.17 => 12.10.14 => 13.0.14 => 13.1.11 => 13.8.8 => 13.12.15 => 14.0.12 => 14.3.6 => 14.9.5 => 14.10.5 => 15.0.5 => 15.4.6 => 15.9.1

This was the commands I done:

gitlab-rake gitlab:storage:migrate_to_hashed
yum install -y gitlab-ce-12.0.12
yum install -y gitlab-ce-12.1.17
yum install -y gitlab-ce-12.10.14
gitlab-rake gitlab:storage:migrate_to_hashed
yum install -y gitlab-ce-13.0.14
yum install -y gitlab-ce-13.1.11
yum install -y gitlab-ce-13.8.8
gitlab-rake gitlab:storage:migrate_to_hashed
yum install -y gitlab-ce-13.12.15
gitlab-rake gitlab:storage:migrate_to_hashed
touch /etc/gitlab/skip-unmigrated-data-check

yum install -y gitlab-ce-14.0.12
yum install -y gitlab-ce-14.3.6

a lot of warnings and failed using yum install -y gitlab-ce-14.3.6:

Warnings:
The version of the running redis service is different than what is installed.
Please restart redis to start the new version.

sudo gitlab-ctl restart redis

The version of the running postgresql service is different than what is installed.
Please restart postgresql to start the new version.

sudo gitlab-ctl restart postgresql

Running handlers complete
Chef Infra Client failed. 30 resources updated in 01 minutes 09 seconds

Warnings:
The version of the running redis service is different than what is installed.
Please restart redis to start the new version.

sudo gitlab-ctl restart redis

The version of the running postgresql service is different than what is installed.
Please restart postgresql to start the new version.

sudo gitlab-ctl restart postgresql

===
There was an error running gitlab-ctl reconfigure. Please check the output above for more
details.

warning: %posttrans(gitlab-ce-14.3.6-ce.0.el7.x86_64) scriptlet failed, exit status 1
Non-fatal POSTTRANS scriptlet failure in rpm package gitlab-ce-14.3.6-ce.0.el7.x86_64
Verifying : gitlab-ce-14.3.6-ce.0.el7.x86_64 1/2
Verifying : gitlab-ce-14.0.12-ce.0.el7.x86_64 2/2

Updated:
gitlab-ce.x86_64 0:14.3.6-ce.0.el7

Complete!

alansenairj · March 3, 2023, 1:19pm

Hi Greg. Let me explain my upgrade process better.

I have a running server using 11.8.2 ee and I resoterd backup of this server in a new clean VM. I am using oracle linux 7.9 there.

I bring back this backup installing 11.8.2 ee and then i upgraded it to 11.11.8 ee.

I use a help site to give me a upgrade path:
12.0.12 => 12.1.17 => 12.10.14 => 13.0.14 => 13.1.11 => 13.8.8 => 13.12.15 => 14.0.12 => 14.3.6 => 14.9.5 => 14.10.5 => 15.0.5 => 15.4.6 => 15.9.1

Now I will write down a sequence of lauched commands I used.

yum install gitlab-ee-11.11.8-ee.0.el7.x86_64.rpm
gitlab-rake gitlab:storage:migrate_to_hashed
yum install -y gitlab-ce-12.0.12
yum install -y gitlab-ce-12.1.17
yum install -y gitlab-ce-12.10.14
gitlab-rake gitlab:storage:migrate_to_hashed
yum install -y gitlab-ce-13.0.14
yum install -y gitlab-ce-13.1.11
yum install -y gitlab-ce-13.8.8
yum install -y gitlab-ce-13.12.15
gitlab-rake gitlab:storage:migrate_to_hashed
touch /etc/gitlab/skip-unmigrated-data-check
yum install -y gitlab-ce-14.0.12
yum install -y gitlab-ce-14.3.6

this last version failed.

Warnings:
The version of the running redis service is different than what is installed.
Please restart redis to start the new version.

sudo gitlab-ctl restart redis

The version of the running postgresql service is different than what is installed.
Please restart postgresql to start the new version.

sudo gitlab-ctl restart postgresql

Running handlers complete
Chef Infra Client failed. 30 resources updated in 01 minutes 09 seconds

Warnings:
The version of the running redis service is different than what is installed.
Please restart redis to start the new version.

sudo gitlab-ctl restart redis

The version of the running postgresql service is different than what is installed.
Please restart postgresql to start the new version.

sudo gitlab-ctl restart postgresql

===
There was an error running gitlab-ctl reconfigure. Please check the output above for more
details.

warning: %posttrans(gitlab-ce-14.3.6-ce.0.el7.x86_64) scriptlet failed, exit status 1
Non-fatal POSTTRANS scriptlet failure in rpm package gitlab-ce-14.3.6-ce.0.el7.x86_64
Verifying : gitlab-ce-14.3.6-ce.0.el7.x86_64 1/2
Verifying : gitlab-ce-14.0.12-ce.0.el7.x86_64 2/2

Updated:
gitlab-ce.x86_64 0:14.3.6-ce.0.el7

Complete!

alansenairj · March 3, 2023, 1:22pm

I restored a VM snapshot to version 13.8.8 ce and I don’t know what can i do to avoid this situation

alansenairj · March 3, 2023, 1:52pm

Same error here:

yum install -y gitlab-ce-13.12.15 && gitlab-rake gitlab:storage:migrate_to_hashed && touch /etc/gitlab/skip-unmigrated-data-check && yum install -y gitlab-ce-14.0.12
yum install -y gitlab-ce-14.0.12
gitlab-rake db:migrate
gitlab-ctl reconfigure
gitlab-ctl restart
yum install -y gitlab-ce-14.3.6

==================================

Warnings:
The version of the running redis service is different than what is installed.
Please restart redis to start the new version.

sudo gitlab-ctl restart redis

Running handlers complete
Chef Infra Client failed. 31 resources updated in 01 minutes 12 seconds

Warnings:
The version of the running redis service is different than what is installed.
Please restart redis to start the new version.

sudo gitlab-ctl restart redis

===
There was an error running gitlab-ctl reconfigure. Please check the output above for more
details.

warning: %posttrans(gitlab-ce-14.3.6-ce.0.el7.x86_64) scriptlet failed, exit status 1
Non-fatal POSTTRANS scriptlet failure in rpm package gitlab-ce-14.3.6-ce.0.el7.x86_64
Verifying : gitlab-ce-14.3.6-ce.0.el7.x86_64 1/2
Verifying : gitlab-ce-14.0.12-ce.0.el7.x86_64 2/2

Updated:
gitlab-ce.x86_64 0:14.3.6-ce.0.el7

Complete!

iwalker · March 3, 2023, 2:20pm

Why are you using that option? You are causing it to ignore potential problems with any unmigrated data.

alansenairj · March 3, 2023, 2:37pm

because of that:

Upgrade 1 Package

Total download size: 869 M
Downloading packages:
Delta RPMs disabled because /usr/bin/applydeltarpm not installed.
gitlab-ce-14.0.12-ce.0.el7.x86_64.rpm | 869 MB 00:01:23
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
gitlab preinstall: Checking for unmigrated data on legacy storage
gitlab preinstall:
gitlab preinstall: Legacy storage is no longer supported. Please migrate your data to hashed storage.
gitlab preinstall: Check Repository storage Rake tasks | GitLab for details.
gitlab preinstall:
gitlab preinstall: If you want to skip this check, run the following command and try again:
gitlab preinstall:
gitlab preinstall: sudo touch /etc/gitlab/skip-unmigrated-data-check
gitlab preinstall:
error: %pre(gitlab-ce-14.0.12-ce.0.el7.x86_64) scriptlet failed, exit status 1
Error in PREIN scriptlet in rpm package gitlab-ce-14.0.12-ce.0.el7.x86_64
gitlab-ce-13.12.15-ce.0.el7.x86_64 was supposed to be removed but is not!
Verifying : gitlab-ce-13.12.15-ce.0.el7.x86_64 1/2
Verifying : gitlab-ce-14.0.12-ce.0.el7.x86_64 2/2

Failed:
gitlab-ce.x86_64 0:13.12.15-ce.0.el7 gitlab-ce.x86_64 0:14.0.12-ce.0.el7

alansenairj · March 3, 2023, 2:39pm

Can you help me to fix this issue?

yum install -y gitlab-ce-13.8.8
442 yum install -y gitlab-ce-13.12.15
443 gitlab-rake gitlab:storage:migrate_to_hashed
444 yum install -y gitlab-ce-14.0.12

It can´t go ahead to 14 if I don’t do this recomended command.

What it does break? How can I avoid it?
The command gitlab-rake gitlab:storage:migrate_to_hashed showed to me an ok result but it is not migrating nothing or it has another problem

alansenairj · March 3, 2023, 2:48pm

As you can see at my history I ran this command before upgrade to 12 and here is it’s output:

[root@gitlab3 ~]# gitlab-rake gitlab:storage:migrate_to_hashed
Enqueuing migration of 209 projects in batches of 200… Done!

iwalker · March 3, 2023, 3:05pm

Well, as from the output it still seems to think you have repositories that haven’t been migrated. So creating the file just ignores it, but that doesn’t solve the problem.

Did you download the RPM’s manually? Or are you using yum/dnf commands for it to download the rpm from Gitlab repository? If you downloaded manually, and then attempted to install, the errors being shown could be because the RPM didn’t download fully and is corrupted. This could be the reason for your problems.

alansenairj · March 3, 2023, 3:10pm

[root@gitlab3 ~]# gitlab-rake gitlab:storage:legacy_projects

Found 209 projects using Legacy Storage

I tried this workaround and it is not migreted too

wget -O /tmp/fix-legacy-hashed-storage-migration.rb https://gitlab.com/snippets/2039252/raw
gitlab-rails runner /tmp/fix-legacy-hashed-storage-migration.rb

alansenairj · March 3, 2023, 3:14pm

I am using rpm from gitlab repo only. I can restore a VM initial state again and run everything but I think it is not migrating correctly. I will do it again but I will try to migrate at 12 version

iwalker · March 3, 2023, 6:51pm

Make sure ALL background migrations have finished before attempting each upgrade. This is important. @gitlab-greg posted that you had some migrations that didn’t finish. So perhaps you were running the upgrades too quick before letting migrations finish?

Your steps should be:

Upgrade Gitlab.
Check background migrations have finished.
Check web interface and make sure all is working. Maybe even do gitlab-ctl restart before attempting next upgrade.
Repeat from step 1 for the next upgrade on the upgrade path.

alansenairj · March 3, 2023, 11:11pm

humm you nailed it. I learned by the hard way. I am working too fast.
I just realize that because new versions has a visual queue at admin/background_migrations

But I got a problem to migrate to hash and I will put I down and how can I solved it.

erro que não consegue migrar para hashed

#https://gitlab3.csivix.com.br/admin/background_jobs = OpenSSL::Cipher::CipherError:

Back up and restore GitLab | GitLab

Problem with background migration - #5 by servuc

gitlab-rake db:migrate
gitlab-rails dbconsole

UPDATE projects SET runners_token = null, runners_token_encrypted = null;
UPDATE namespaces SET runners_token = null, runners_token_encrypted = null;
UPDATE application_settings SET runners_registration_token_encrypted = null;
UPDATE application_settings SET encrypted_ci_jwt_signing_key = null;
UPDATE ci_runners SET token = null, token_encrypted = null;
UPDATE ci_builds SET token = null, token_encrypted = null;

ctrl+d

gitlab-rake gitlab:storage:migrate_to_hashed
gitlab-rake gitlab:storage:hashed_projects
gitlab-rake gitlab:storage:legacy_projects

gitlab-ctl restart

Migration path is not working

=== There was an error running gitlab-ctl reconfigure. Please check the output above for more details.

=== There was an error running gitlab-ctl reconfigure. Please check the output above for more details.

=== There was an error running gitlab-ctl reconfigure. Please check the output above for more details.

erro que não consegue migrar para hashed

Back up and restore GitLab | GitLab

Problem with background migration - #5 by servuc

===
There was an error running gitlab-ctl reconfigure. Please check the output above for more
details.

===
There was an error running gitlab-ctl reconfigure. Please check the output above for more
details.

===
There was an error running gitlab-ctl reconfigure. Please check the output above for more
details.