Migration 14.5 to 14.6 failed with migrating EncryptStaticObjectsExternal StorageAuthToken

Hi there. Unfortunately the last upgrade rendered our GitLab installation completely unusable.

On first running apt upgrade the installation complained about running a wrong redis version and that I should restart redis, what I did. During the second run a migration failed and there are not hints whatsoever on how to fix it.

As the output of the upgrade script was truncated I ran gitlab-rake db:migrate --trace which gave a bit more information about the error but wasn’t of any more help to me to find a solution.

== 20211126115449 EncryptStaticObjectsExternalStorageAuthToken: migrating =====
rake aborted!
StandardError: An error has occurred, this and all later migrations canceled:

data must not be empty

[...]

Caused by:
ArgumentError: data must not be empty
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/encryptor-3.0.0/lib/encryptor.rb:97:in `update'

Full output is below.

Does anyone have an idea? I googled, searched the issues for both gitlab and omnibus and checked this forum but did not find a solution…

Summary
** Invoke db:migrate (first_time)
** Invoke db:load_config (first_time)
** Invoke environment (first_time)
** Execute environment
** Execute db:load_config
** Invoke schema_version_check (first_time)
** Invoke environment
** Execute schema_version_check
** Execute db:migrate
== 20211126115449 EncryptStaticObjectsExternalStorageAuthToken: migrating =====
rake aborted!
StandardError: An error has occurred, this and all later migrations canceled:

data must not be empty
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/encryptor-3.0.0/lib/encryptor.rb:97:in `update'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/encryptor-3.0.0/lib/encryptor.rb:97:in `crypt'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/encryptor-3.0.0/lib/encryptor.rb:36:in `encrypt'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/crypto_helper.rb:20:in `aes256_gcm_encrypt'
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20211126115449_encrypt_static_objects_external_storage_auth_token.rb:16:in `block in up'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/relation/batches.rb:71:in `block (2 levels) in find_each'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/relation/batches.rb:71:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/relation/batches.rb:71:in `block in find_each'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/relation/batches.rb:138:in `block in find_in_batches'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/relation/batches.rb:245:in `block in in_batches'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/relation/batches.rb:229:in `loop'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/relation/batches.rb:229:in `in_batches'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/relation/batches.rb:137:in `find_in_batches'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/relation/batches.rb:70:in `find_each'
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20211126115449_encrypt_static_objects_external_storage_auth_token.rb:15:in `up'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/migration.rb:870:in `public_send'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/migration.rb:870:in `exec_migration'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/migration.rb:851:in `block (2 levels) in migrate'
/opt/gitlab/embedded/lib/ruby/2.7.0/benchmark.rb:293:in `measure'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/migration.rb:850:in `block in migrate'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:462:in `with_connection'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/migration.rb:849:in `migrate'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/migration.rb:1037:in `migrate'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/migration.rb:1329:in `block in execute_migration_in_transaction'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/migration.rb:1380:in `block in ddl_transaction'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/connection_adapters/abstract/database_statements.rb:320:in `block in transaction'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/connection_adapters/abstract/transaction.rb:319:in `block in within_new_transaction'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.1.4.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in `block (2 levels) in synchronize'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.1.4.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.1.4.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.1.4.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.1.4.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/connection_adapters/abstract/transaction.rb:317:in `within_new_transaction'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/connection_adapters/abstract/database_statements.rb:320:in `transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `block in write_using_load_balancer'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:112:in `block in read_write'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:172:in `retry_with_backoff'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:110:in `read_write'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:125:in `write_using_load_balancer'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:77:in `transaction'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/transactions.rb:209:in `transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database.rb:265:in `block in transaction'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.1.4.1/lib/active_support/notifications.rb:205:in `instrument'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database.rb:264:in `transaction'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/migration.rb:1380:in `ddl_transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:31:in `ddl_transaction'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/migration.rb:1328:in `execute_migration_in_transaction'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/migration.rb:1302:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/migration.rb:1302:in `migrate_without_lock'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/migration.rb:1251:in `block in migrate'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/migration.rb:1401:in `block in with_advisory_lock'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/migration.rb:1416:in `block in with_advisory_lock_connection'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:462:in `with_connection'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/migration.rb:1416:in `with_advisory_lock_connection'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/migration.rb:1397:in `with_advisory_lock'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/migration.rb:1251:in `migrate'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/migration.rb:1086:in `up'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/migration.rb:1061:in `migrate'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/tasks/database_tasks.rb:237:in `migrate'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/railties/databases.rake:92:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/railties/databases.rake:90:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/railties/databases.rake:90:in `block (2 levels) in <top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `block in execute'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `execute'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:219:in `block in invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `synchronize'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:188:in `invoke'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:160:in `invoke_task'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `block (2 levels) in top_level'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `block in top_level'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:125:in `run_with_threads'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:110:in `top_level'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:83:in `block in run'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:186:in `standard_exception_handling'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:80:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/exe/rake:27:in `<top (required)>'
/opt/gitlab/embedded/bin/rake:23:in `load'
/opt/gitlab/embedded/bin/rake:23:in `<top (required)>'
/opt/gitlab/embedded/lib/ruby/site_ruby/2.7.0/bundler/cli/exec.rb:63:in `load'
/opt/gitlab/embedded/lib/ruby/site_ruby/2.7.0/bundler/cli/exec.rb:63:in `kernel_load'
/opt/gitlab/embedded/lib/ruby/site_ruby/2.7.0/bundler/cli/exec.rb:28:in `run'
/opt/gitlab/embedded/lib/ruby/site_ruby/2.7.0/bundler/cli.rb:476:in `exec'
/opt/gitlab/embedded/lib/ruby/site_ruby/2.7.0/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
/opt/gitlab/embedded/lib/ruby/site_ruby/2.7.0/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
/opt/gitlab/embedded/lib/ruby/site_ruby/2.7.0/bundler/vendor/thor/lib/thor.rb:399:in `dispatch'
/opt/gitlab/embedded/lib/ruby/site_ruby/2.7.0/bundler/cli.rb:30:in `dispatch'
/opt/gitlab/embedded/lib/ruby/site_ruby/2.7.0/bundler/vendor/thor/lib/thor/base.rb:476:in `start'
/opt/gitlab/embedded/lib/ruby/site_ruby/2.7.0/bundler/cli.rb:24:in `start'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/bundler-2.1.4/exe/bundle:46:in `block in <top (required)>'
/opt/gitlab/embedded/lib/ruby/site_ruby/2.7.0/bundler/friendly_errors.rb:123:in `with_friendly_errors'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/bundler-2.1.4/exe/bundle:34:in `<top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'

Caused by:
ArgumentError: data must not be empty
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/encryptor-3.0.0/lib/encryptor.rb:97:in `update'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/encryptor-3.0.0/lib/encryptor.rb:97:in `crypt'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/encryptor-3.0.0/lib/encryptor.rb:36:in `encrypt'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/crypto_helper.rb:20:in `aes256_gcm_encrypt'
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20211126115449_encrypt_static_objects_external_storage_auth_token.rb:16:in `block in up'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/relation/batches.rb:71:in `block (2 levels) in find_each'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/relation/batches.rb:71:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/relation/batches.rb:71:in `block in find_each'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/relation/batches.rb:138:in `block in find_in_batches'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/relation/batches.rb:245:in `block in in_batches'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/relation/batches.rb:229:in `loop'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/relation/batches.rb:229:in `in_batches'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/relation/batches.rb:137:in `find_in_batches'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/relation/batches.rb:70:in `find_each'
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20211126115449_encrypt_static_objects_external_storage_auth_token.rb:15:in `up'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/migration.rb:870:in `public_send'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/migration.rb:870:in `exec_migration'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/migration.rb:851:in `block (2 levels) in migrate'
/opt/gitlab/embedded/lib/ruby/2.7.0/benchmark.rb:293:in `measure'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/migration.rb:850:in `block in migrate'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:462:in `with_connection'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/migration.rb:849:in `migrate'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/migration.rb:1037:in `migrate'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/migration.rb:1329:in `block in execute_migration_in_transaction'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/migration.rb:1380:in `block in ddl_transaction'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/connection_adapters/abstract/database_statements.rb:320:in `block in transaction'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/connection_adapters/abstract/transaction.rb:319:in `block in within_new_transaction'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.1.4.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in `block (2 levels) in synchronize'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.1.4.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.1.4.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.1.4.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.1.4.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/connection_adapters/abstract/transaction.rb:317:in `within_new_transaction'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/connection_adapters/abstract/database_statements.rb:320:in `transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `block in write_using_load_balancer'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:112:in `block in read_write'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:172:in `retry_with_backoff'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:110:in `read_write'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:125:in `write_using_load_balancer'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:77:in `transaction'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/transactions.rb:209:in `transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database.rb:265:in `block in transaction'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.1.4.1/lib/active_support/notifications.rb:205:in `instrument'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database.rb:264:in `transaction'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/migration.rb:1380:in `ddl_transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:31:in `ddl_transaction'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/migration.rb:1328:in `execute_migration_in_transaction'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/migration.rb:1302:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/migration.rb:1302:in `migrate_without_lock'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/migration.rb:1251:in `block in migrate'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/migration.rb:1401:in `block in with_advisory_lock'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/migration.rb:1416:in `block in with_advisory_lock_connection'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:462:in `with_connection'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/migration.rb:1416:in `with_advisory_lock_connection'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/migration.rb:1397:in `with_advisory_lock'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/migration.rb:1251:in `migrate'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/migration.rb:1086:in `up'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/migration.rb:1061:in `migrate'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/tasks/database_tasks.rb:237:in `migrate'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/railties/databases.rake:92:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/railties/databases.rake:90:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/railties/databases.rake:90:in `block (2 levels) in <top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `block in execute'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `execute'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:219:in `block in invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `synchronize'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:188:in `invoke'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:160:in `invoke_task'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `block (2 levels) in top_level'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `block in top_level'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:125:in `run_with_threads'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:110:in `top_level'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:83:in `block in run'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:186:in `standard_exception_handling'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:80:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/exe/rake:27:in `<top (required)>'
/opt/gitlab/embedded/bin/rake:23:in `load'
/opt/gitlab/embedded/bin/rake:23:in `<top (required)>'
/opt/gitlab/embedded/lib/ruby/site_ruby/2.7.0/bundler/cli/exec.rb:63:in `load'
/opt/gitlab/embedded/lib/ruby/site_ruby/2.7.0/bundler/cli/exec.rb:63:in `kernel_load'
/opt/gitlab/embedded/lib/ruby/site_ruby/2.7.0/bundler/cli/exec.rb:28:in `run'
/opt/gitlab/embedded/lib/ruby/site_ruby/2.7.0/bundler/cli.rb:476:in `exec'
/opt/gitlab/embedded/lib/ruby/site_ruby/2.7.0/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
/opt/gitlab/embedded/lib/ruby/site_ruby/2.7.0/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
/opt/gitlab/embedded/lib/ruby/site_ruby/2.7.0/bundler/vendor/thor/lib/thor.rb:399:in `dispatch'
/opt/gitlab/embedded/lib/ruby/site_ruby/2.7.0/bundler/cli.rb:30:in `dispatch'
/opt/gitlab/embedded/lib/ruby/site_ruby/2.7.0/bundler/vendor/thor/lib/thor/base.rb:476:in `start'
/opt/gitlab/embedded/lib/ruby/site_ruby/2.7.0/bundler/cli.rb:24:in `start'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/bundler-2.1.4/exe/bundle:46:in `block in <top (required)>'
/opt/gitlab/embedded/lib/ruby/site_ruby/2.7.0/bundler/friendly_errors.rb:123:in `with_friendly_errors'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/bundler-2.1.4/exe/bundle:34:in `<top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => db:migrate

I restored the installation from backup. This time it did not complain about redis but the installation still failed while migrating.

== 20211126115449 EncryptStaticObjectsExternalStorageAuthToken: migrating =====
STDERR:
---- End output of "bash"  "/tmp/chef-script20211223-13864-ts3c33" ----
Ran "bash"  "/tmp/chef-script20211223-13864-ts3c33" returned 1

Running handlers complete
Chef Infra Client failed. 14 resources updated in 45 seconds
===
There was an error running gitlab-ctl reconfigure. Please check the output above for more
details.
===

dpkg: error processing package gitlab-ee (--configure):
 installed gitlab-ee package post-installation script subprocess returned error exit status 1
Errors were encountered while processing:
 gitlab-ee
E: Sub-process /usr/bin/dpkg returned an error code (1)

Hello @wedibit , did you manage to resolve this issue? I have the same problem too. Thanks

No. I tried all workarounds I found but nothing helped. In the end I reverted to a backup and postponed further work on this problem until the next necessary update.

I had this issue as well. Just wanted to share how I fixed it.

I looked at the migration code that was added that is causing the issue Encrypt plaintext static_objects_external_storage_auth_token fields (!75417) · Merge requests · GitLab.org / GitLab · GitLab

I believe what happened is in the gitlab application_settings postgres table, the static_objects_external_storage_auth_token and static_objects_external_storage_auth_token_encrypted columns at some point were not initialized as null (this code is looking for null).

I used this command to get into the database on my omnibus server

sudo -u gitlab-psql /opt/gitlab/embedded/bin/psql -h /var/opt/gitlab/postgresql/ gitlabhq_production

Then I could select the two columns it is using with this:

gitlabhq_production=# SELECT static_objects_external_storage_auth_token, static_objects_external_storage_auth_token_encrypted FROM application_settings;
 static_objects_external_storage_auth_token | static_objects_external_storage_auth_token_encrypted 
--------------------------------------------+------------------------------------------------------
                                            | 
(1 row)

I couldn’t see what the data types were from the console, however since they were blank for my instance (I’m guessing empty string/text type column from bad initialization), I just ran two update queries to set them to NULL instead.

UPDATE application_settings SET static_objects_external_storage_auth_token = NULL;
UPDATE application_settings SET static_objects_external_storage_auth_token_encrypted = NULL;

After I did that, I typed exit to get out of the database scope, then ran sudo gitlab-ctl reconfigure and everything worked as expected.

Hope this helps someone.

2 Likes

Hey there,

I’ve created an issue on gitlab.com since we face this error, too.

Hi @markaius

I’ve followed your instructions and it worked! Thank you!

1 Like

Hi @markaius,

thanks for your instructions, i used them with my gitlab with docker, and i was able to start version 14.6.1.
Complete for those who use docker, to access the container (with the working and previous version)

docker exec -u gitlab-psql -it <gitlab-container> bash

start the command without sudo -u gitlab-psql

/opt/gitlab/embedded/bin/psql -h /var/opt/gitlab/postgresql/ gitlabhq_production

Continue to follow the steps indicated by @markaius.

Get out of container and start the container with the new version.

I hope this helps someone else :slight_smile:

1 Like