Upgrade from 16.3.5 - 16.4.1 failing

michael.adriaanse · October 3, 2023, 10:02am

Hi guys,

I need some assistance please. When running a gitlab:check, I’m receiving the following errors.

GitLab Shell: … GitLab Shell version >= 14.28.0 ? … OK (14.28.0)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Internal API available: FAILED - Internal API unreachable
gitlab-shell self-check failed
Try fixing it:
Make sure GitLab is running;
Check the gitlab-shell configuration file:
sudo -u git -H editor /opt/gitlab/embedded/service/gitlab-shell/config.yml
Please fix the error above and rerun the checks.

All migrations up? … no
Try fixing it:
sudo -u git -H bundle exec rake db:migrate RAILS_ENV=production
Please fix the error above and rerun the checks.

I haven’t changed anything when upgrading and really not sure what the issue is.

balonik · October 4, 2023, 7:33am

Hi, have you tried to perform the suggested steps?

michael.adriaanse · October 4, 2023, 1:41pm

Hi,

I’m getting this output when running the migrate command. I haven’t really found anything helpful yet.
root@testgit:/gitlab# docker exec -it gitlab-gitlab-1 sh -c ‘gitlab-rake db:migrate RAILS_ENV=production’

main: == [advisory_lock_connection] object_id: 169600, pg_backend_pid: 300

main: == 20230622073752 EnsureBackfillCiPipelinesBigintIdIsCompleted: migrating =====

main: – transaction_open?()

main: → 0.0000s

main: – transaction_open?()

main: → 0.0000s

rake aborted!

StandardError: An error has occurred, all later migrations canceled:

PG::IndexCorrupted: ERROR: table tid from new index tuple (400,6) overlaps with invalid duplicate tuple at offset 6 of block 14 in index “index_ci_pipelines_on_project_id_and_user_id_and_status_and_ref”

/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `public_send’

/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `block in write_using_load_balancer’

/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:133:in `block in read_write’

/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:204:in `retry_with_backoff’

/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:122:in `read_write’

/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `write_using_load_balancer’

/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:61:in `block (2 levels) in class:ConnectionProxy’

/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration/copy_column_using_background_migration_job.rb:25:in `block in perform’

/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration/batched_migration_job.rb:105:in `block (2 levels) in each_sub_batch’

/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batch_metrics.rb:22:in `instrument_operation’

/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration/batched_migration_job.rb:104:in `block in each_sub_batch’

/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/each_batch.rb:99:in `block (2 levels) in each_batch’

/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/each_batch.rb:99:in `block in each_batch’

/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/each_batch.rb:69:in `step’

/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/each_batch.rb:69:in `each_batch’

/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration/batched_migration_job.rb:103:in `each_sub_batch’

/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration/copy_column_using_background_migration_job.rb:24:in `perform’

/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_wrapper.rb:81:in `execute_batched_migration_job’

/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_wrapper.rb:63:in `execute_job’

/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_wrapper.rb:50:in `execute_batch’

/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_wrapper.rb:25:in `perform’

/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_runner.rb:30:in `run_migration_job’

/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_runner.rb:140:in `run_migration_while’

/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_runner.rb:80:in `finalize’

/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_runner.rb:10:in `finalize’

/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/batched_background_migration_helpers.rb:160:in `block (2 levels) in finalize_batched_background_migration’

/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/query_analyzers/base.rb:39:in `with_suppressed’

/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/batched_background_migration_helpers.rb:159:in `block in finalize_batched_background_migration’

/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/reestablished_connection_stack.rb:21:in `with_restored_connection_stack’

/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/batched_background_migration_helpers.rb:158:in `finalize_batched_background_migration’

/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/batched_background_migration_helpers.rb:231:in `ensure_batched_background_migration_is_finished’

/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20230622073752_ensure_backfill_ci_pipelines_bigint_id_is_completed.rb:12:in `up’

/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:33:in `block in exec_migration’

/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/query_analyzer.rb:37:in `within’

/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:30:in `exec_migration’

/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/automatic_lock_writes_on_tables.rb:21:in `exec_migration’

/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:36:in `ddl_transaction’

/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/runner_backoff/active_record_mixin.rb:21:in `execute_migration_in_transaction’

/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:15:in `block in with_advisory_lock_connection’

/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:12:in `with_advisory_lock_connection’

/opt/gitlab/embedded/bin/bundle:25:in `load’

/opt/gitlab/embedded/bin/bundle:25:in `’

Caused by:

ActiveRecord::StatementInvalid: PG::IndexCorrupted: ERROR: table tid from new index tuple (400,6) overlaps with invalid duplicate tuple at offset 6 of block 14 in index “index_ci_pipelines_on_project_id_and_user_id_and_status_and_ref”