We upgraded from EE v13.12.15 to EE v14.0.12. We made sure that all background migrations and batched background migrations completed successfully. Confirmed in Gitlab Admin → Monitoring → Background Migrations (showed 14 finished, 0 queues, 0 failures) and running
sudo gitlab-rails runner -e production ‘puts Gitlab::BackgroundMigration.remaining’
and
sudo gitlab-rails runner -e production ‘puts Gitlab::Database::BackgroundMigration::BatchedMigration.queued.count’
and
sudo gitlab-rails runner -e production ‘Gitlab::Database::BackgroundMigration::BatchedMigration.failed.count’

All came up with ‘0’. So we proceeded with the upgrade to EE v14.3.6. In the logs, we see this:
Recipe: gitlab::gitlab-rails

• execute[clear the gitlab-rails cache] action run

  • execute /opt/gitlab/bin/gitlab-rake cache:clear
    Recipe: gitaly::enable

• runit_service[gitaly] action restart

  ================================================================================
  Error executing action restart on resource ‘runit_service[gitaly]’

  Mixlib::ShellOut::ShellCommandFailed

  Expected process to exit with [0], but received ‘1’
  ---- Begin output of /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/gitaly ----
  STDOUT: timeout: run: /opt/gitlab/service/gitaly: (pid 15622) 677s, got TERM
  STDERR:
  ---- End output of /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/gitaly ----
  Ran /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/gitaly returned 1

And

Running handlers:
There was an error running gitlab-ctl reconfigure:

runit_service[gitaly] (gitaly::enable line 103) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received ‘1’
---- Begin output of /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/gitaly ----
STDOUT: timeout: run: /opt/gitlab/service/gitaly: (pid 15622) 677s, got TERM
STDERR:
---- End output of /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/gitaly ----
Ran /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/gitaly returned 1

Warnings:
The version of the running redis service is different than what is installed.
Please restart redis to start the new version.

sudo gitlab-ctl restart redis

Running handlers complete
Chef Infra Client failed. 45 resources updated in 01 minutes 55 seconds

Warnings:
The version of the running redis service is different than what is installed.
Please restart redis to start the new version.

sudo gitlab-ctl restart redis

===
There was an error running gitlab-ctl reconfigure. Please check the output above for more
details.

warning: %posttrans(gitlab-ee-14.3.6-ee.0.el8.x86_64) scriptlet failed, exit status 1

Error in POSTTRANS scriptlet in rpm package gitlab-ee
Running scriptlet: gitlab-ee-14.0.12-ee.0.el8.x86_64

We ran gitlab-ctl stop once we were sure no more upgrade processing was occurring. We ran gitlab-ctl stop and then gitlab-reconfigure and then gitlab-ctl start.

Everything comes up, and in the application I see the versions all look correct . The Monitoring → Background Migrations showed 1 new Finished Background Migration and 0 queued and 0 failed. But when we run
sudo gitlab-rails runner -e production ‘puts Gitlab::BackgroundMigration.remaining’

It shows 2 remaining.

Any help would be appreciated!

We did a flashback to v14.0.12 and then first upgraded to 14.1.8, then 14.2.7. These completed successfully. Then we tried 14.3.6 again. This time we saw similar messages in the logs, but the background migrations queued count finally showed 0 at the OS level after some time. We were then able to continue on with updates to 14.6.2, 14.9.5 and finally 14.10.5.