Gitlab 14.5.3 database migrations fails after update from 14.4.5

Hi, first time poster here.

We are running into problems when we are doing upgrade from 14.4.5 to 14.5.3.

Problems arise in gitlab-rake db:migrate phase when it hits this stage:
== 20211012134316 CleanUpMigrateMergeRequestDiffCommitUsers: migrating ========
– 21 MigrateMergeRequestDiffCommitUsers are still pending

Currently we are trying to migrate this with virtual machine that has 256 GB memory so that shouldn’t be an issue.

We checked the size of merge_request_diff_commits table and it’s about 175197110 lines and about 53 GB.

It fails after running about 17-18 hours with following error message:

rake aborted!
StandardError: An error has occurred, this and all later migrations canceled:

PG::UnableToSend: no connection to the server
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:126:in block in write_using_load_balancer' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:112:in block in read_write’
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:172:in retry_with_backoff' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:110:in read_write’
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:125:in write_using_load_balancer' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:77:in transaction’
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database.rb:252:in block in transaction' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database.rb:251:in transaction’
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:31:in ddl_transaction' /opt/gitlab/embedded/bin/bundle:23:in load’
/opt/gitlab/embedded/bin/bundle:23:in `’

Caused by:
ActiveRecord::StatementInvalid: PG::UnableToSend: no connection to the server
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:126:in block in write_using_load_balancer' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:112:in block in read_write’
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:172:in retry_with_backoff' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:110:in read_write’
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:125:in write_using_load_balancer' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:77:in transaction’
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database.rb:252:in block in transaction' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database.rb:251:in transaction’
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:31:in ddl_transaction' /opt/gitlab/embedded/bin/bundle:23:in load’
/opt/gitlab/embedded/bin/bundle:23:in `’

Caused by:
PG::UnableToSend: no connection to the server
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:126:in block in write_using_load_balancer' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:112:in block in read_write’
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:172:in retry_with_backoff' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:110:in read_write’
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:125:in write_using_load_balancer' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:77:in transaction’
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database.rb:252:in block in transaction' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database.rb:251:in transaction’
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:31:in ddl_transaction' /opt/gitlab/embedded/bin/bundle:23:in load’
/opt/gitlab/embedded/bin/bundle:23:in `’

Caused by:
ActiveRecord::StatementInvalid: PG::UnableToSend: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:106:in block in read_using_load_balancer' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:112:in block in read_write’
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:172:in retry_with_backoff' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:110:in read_write’
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:105:in read_using_load_balancer' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:54:in select_all’
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration/migrate_merge_request_diff_commit_users.rb:65:in block in each_row_to_migrate' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/pagination/keyset/iterator.rb:25:in block in each_batch’
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/pagination/keyset/iterator.rb:19:in loop' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/pagination/keyset/iterator.rb:19:in each_batch’
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration/migrate_merge_request_diff_commit_users.rb:65:in each_row_to_migrate' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration/migrate_merge_request_diff_commit_users.rb:119:in get_data_to_update’
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration/migrate_merge_request_diff_commit_users.rb:87:in perform' /opt/gitlab/embedded/service/gitlab-rails/db/migrate/20211012134316_clean_up_migrate_merge_request_diff_commit_users.rb:29:in block in up’
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20211012134316_clean_up_migrate_merge_request_diff_commit_users.rb:26:in each' /opt/gitlab/embedded/service/gitlab-rails/db/migrate/20211012134316_clean_up_migrate_merge_request_diff_commit_users.rb:26:in up’
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:126:in block in write_using_load_balancer' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:112:in block in read_write’
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:172:in retry_with_backoff' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:110:in read_write’
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:125:in write_using_load_balancer' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:77:in transaction’
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database.rb:252:in block in transaction' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database.rb:251:in transaction’
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:31:in ddl_transaction' /opt/gitlab/embedded/bin/bundle:23:in load’
/opt/gitlab/embedded/bin/bundle:23:in `’

Caused by:
PG::UnableToSend: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:106:in block in read_using_load_balancer' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:112:in block in read_write’
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:172:in retry_with_backoff' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:110:in read_write’
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:105:in read_using_load_balancer' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:54:in select_all’
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration/migrate_merge_request_diff_commit_users.rb:65:in block in each_row_to_migrate' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/pagination/keyset/iterator.rb:25:in block in each_batch’
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/pagination/keyset/iterator.rb:19:in loop' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/pagination/keyset/iterator.rb:19:in each_batch’
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration/migrate_merge_request_diff_commit_users.rb:65:in each_row_to_migrate' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration/migrate_merge_request_diff_commit_users.rb:119:in get_data_to_update’
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration/migrate_merge_request_diff_commit_users.rb:87:in perform' /opt/gitlab/embedded/service/gitlab-rails/db/migrate/20211012134316_clean_up_migrate_merge_request_diff_commit_users.rb:29:in block in up’
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20211012134316_clean_up_migrate_merge_request_diff_commit_users.rb:26:in each' /opt/gitlab/embedded/service/gitlab-rails/db/migrate/20211012134316_clean_up_migrate_merge_request_diff_commit_users.rb:26:in up’
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:126:in block in write_using_load_balancer' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:112:in block in read_write’
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:172:in retry_with_backoff' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:110:in read_write’
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:125:in write_using_load_balancer' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:77:in transaction’
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database.rb:252:in block in transaction' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database.rb:251:in transaction’
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:31:in ddl_transaction' /opt/gitlab/embedded/bin/bundle:23:in load’
/opt/gitlab/embedded/bin/bundle:23:in `’
Tasks: TOP => db:migrate
(See full trace by running task with --trace)

Anyone?

Removed postgresql timeouts from runtime.conf and now it has gone over 28h.

Table merge_request_diff_commits has grown now over 67GB
gitlabhq_production=# \dt+ merge_request_diff_commits
List of relations
Schema | Name | Type | Owner | Size | Description
--------±---------------------------±------±-------±------±------------
public | merge_request_diff_commits | table | gitlab | 67 GB |
(1 row)

So it seems that it is still gathering data to it before dropping 4 columns (author_name, author_email, committer_name, committer_email) on the next step.

So we finally managed to get our first clone environment migrated last weekend and another clone environment this week so we did get prediction of the time it takes to upgrade our primary gitlab environment. It took over 32h to migrate data to table merge_request_diff_commits.

So disable automatic gitlab-db upgrade from gitlab.rb. Run the gitlab-rake db:migrate on screen app so network etc. timeouts don’t cause problems and lastly disable postgresql timeouts from runtime.conf.

runtime.conf timeouts were:
runtime.conf:statement_timeout = 60000
runtime.conf:idle_in_transaction_session_timeout = 60000

Next step is to plan gitlab update/upgrade session for some weekend and hopefully get everything up-to-date.