Gitlab 14.0.11 Kubernetes deployment problem -- migrations fail with pre-populated database

Hello.

I’m trying to deploy GitLab 14.0.11 (using official gitlab helm chart version 5.0.11) instance in Kubernetes. I already have test GitLab-omnibus installation (which is also 14.0.11) running on EC2 instance in AWS with RDS database. What I want to achieve is to run GitLab in Kubernetes with my existing SQL data, so I stop my test GitLab instance, copied its database using CREATE DATABASE gitlab_k8s TEMPLATE gitlab_test; and configure GitLab kubernetes deployment to use it.

My RDS instance is PostgreSQL 12.7 which is compatible with GitLab 14.x

Unfortunately gitlab-migrations-1 Job is failing. Here is a log snippet from pod:

certificates WARNING: ca-certificates.crt does not contain exactly one certificate or CRL: skipping                                                                                                                 
configure '/init-config/rails-secrets/./secrets.yml' -> '/init-secrets/rails-secrets/./secrets.yml'                                                                                                                 
configure '/init-config/rails-secrets/.' -> '/init-secrets/rails-secrets/.'                                                                                                                                         
configure '/init-config/migrations/./initial_root_password' -> '/init-secrets/migrations/./initial_root_password'                                                                                                   
configure '/init-config/migrations/./gitlab_shared_runners_registration_token' -> '/init-secrets/migrations/./gitlab_shared_runners_registration_token'                                                             
configure '/init-config/migrations/.' -> '/init-secrets/migrations/.'                                                                                                                                               
configure '/init-config/gitaly/./gitaly_token' -> '/init-secrets/gitaly/./gitaly_token'                                                                                                                             
configure '/init-config/gitaly/.' -> '/init-secrets/gitaly/.'                                                                                                                                                       
configure '/init-config/redis/./redis-password' -> '/init-secrets/redis/./redis-password'                                                                                                                           
configure '/init-config/redis/.' -> '/init-secrets/redis/.'                                                                                                                                                         
configure '/init-config/postgres/./psql-password' -> '/init-secrets/postgres/./psql-password'                                                                                                                       
configure '/init-config/postgres/.' -> '/init-secrets/postgres/.'                                                                                                                                                   
migrations Begin parsing .erb files from /var/opt/gitlab/templates                                                                                                                                                  
migrations Writing /srv/gitlab/config/resque.yml                                                                                                                                                                    
migrations Writing /srv/gitlab/config/cable.yml                                                                                                                                                                     
migrations Writing /srv/gitlab/config/database.yml                                                                                                                                                                  
migrations Writing /srv/gitlab/config/gitlab.yml                                                                                                                                                                    
migrations Copying other config files found in /var/opt/gitlab/templates                                                                                                                                            
migrations Attempting to run '/scripts/wait-for-deps /scripts/db-migrate' as a main process                                                                                                                         
migrations #<Thread:0x00007f59fa862918 /scripts/rails-dependencies:16 run> terminated with exception (report_on_exception is true):                                                                                 
migrations /scripts/lib/checks/postgresql.rb:58:in `database_schema_version': undefined method `database' for #<Hash:0x00007f59fb501a98> (NoMethodError)                                                            
migrations     from /scripts/lib/checks/postgresql.rb:72:in `check_schema_version'                                                                                                                                  
migrations     from /scripts/lib/checks/postgresql.rb:17:in `run'                                                                                                                                                   
migrations     from /scripts/rails-dependencies:17:in `block in <main>'                                                                                                                                             
migrations /scripts/lib/checks/postgresql.rb:58:in `database_schema_version': undefined method `database' for #<Hash:0x00007f59fb501a98> (NoMethodError)                                                            
migrations     from /scripts/lib/checks/postgresql.rb:72:in `check_schema_version'                                                                                                                                  
migrations     from /scripts/lib/checks/postgresql.rb:17:in `run'                                                                                                                                                   
migrations     from /scripts/rails-dependencies:17:in `block in <main>'                                                                                                                                             
migrations Checking: resque.yml, cable.yml                                                                                                                                                                          
migrations + SUCCESS connecting to 'redis://gitlab-redis-master.gitlab.svc:6379' from cable.yml, through gitlab-redis-master.gitlab.svc                                                                             
migrations + SUCCESS connecting to 'redis://gitlab-redis-master.gitlab.svc:6379' from resque.yml, through gitlab-redis-master.gitlab.svc

I appreciate any help with this.

Seems that I incorrectly started my deployment without database. Now after I verified that database is in place I see following error related to OpenSSL:

migrations Performing custom instance setup                                                                                                                                                                         
migrations /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/encryptor-3.0.0/lib/encryptor.rb:98:in `final': OpenSSL::Cipher::CipherError                                                                                   
migrations     from /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/encryptor-3.0.0/lib/encryptor.rb:98:in `crypt'                                                                                                        
migrations     from /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/encryptor-3.0.0/lib/encryptor.rb:49:in `decrypt'                                                                                                      
migrations     from /srv/gitlab/lib/gitlab/crypto_helper.rb:28:in `aes256_gcm_decrypt'                                                                                                                              
migrations     from /srv/gitlab/app/models/concerns/token_authenticatable_strategies/encryption_helper.rb:18:in `decrypt_token'                                                                                     
migrations     from /srv/gitlab/app/models/concerns/token_authenticatable_strategies/encrypted.rb:45:in `get_token'                                                                                                 
migrations     from /srv/gitlab/app/models/concerns/token_authenticatable_strategies/base.rb:32:in `ensure_token'                                                                                                   
migrations     from /srv/gitlab/app/models/concerns/token_authenticatable_strategies/encrypted.rb:32:in `ensure_token'                                                                                              
migrations     from /srv/gitlab/app/models/concerns/token_authenticatable.rb:43:in `block in add_authentication_token_field'                                                                                        
migrations     from /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.3.2/lib/active_support/callbacks.rb:427:in `block in make_lambda'                                                                   
migrations     from /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.3.2/lib/active_support/callbacks.rb:198:in `block (2 levels) in halting'                                                            
migrations     from /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.3.2/lib/active_support/callbacks.rb:604:in `block (2 levels) in default_terminator'                                                 
migrations     from /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.3.2/lib/active_support/callbacks.rb:603:in `catch'                                                                                  
migrations     from /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.3.2/lib/active_support/callbacks.rb:603:in `block in default_terminator'                                                            
migrations     from /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.3.2/lib/active_support/callbacks.rb:199:in `block in halting'                                                                       
migrations     from /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.3.2/lib/active_support/callbacks.rb:512:in `block in invoke_before'                                                                 
migrations     from /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.3.2/lib/active_support/callbacks.rb:512:in `each'                                                                                   
migrations     from /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.3.2/lib/active_support/callbacks.rb:512:in `invoke_before'                                                                          
migrations     from /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.3.2/lib/active_support/callbacks.rb:105:in `run_callbacks'                                                                          
migrations     from /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.3.2/lib/active_support/callbacks.rb:824:in `_run_save_callbacks'                                                                    
migrations     from /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/callbacks.rb:457:in `create_or_update'                                                                         
migrations     from /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/timestamp.rb:126:in `create_or_update'                                                                         
migrations     from /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/persistence.rb:474:in `save'                                                                                   
migrations     from /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/validations.rb:47:in `save'                                                                                    
migrations     from /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/transactions.rb:298:in `block in save'                                                                         
migrations     from /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/transactions.rb:354:in `block in with_transaction_returning_status'                                            
migrations     from /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/connection_adapters/abstract/database_statements.rb:320:in `block in transaction'                              
migrations     from /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/connection_adapters/abstract/transaction.rb:310:in `block in within_new_transaction'                           
migrations     from /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.3.2/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in `block (2 levels) in synchronize'                          
migrations     from /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.3.2/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'                                         
migrations     from /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.3.2/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'                                     
migrations     from /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.3.2/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'                                         
migrations     from /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.3.2/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'                                              
migrations     from /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/connection_adapters/abstract/transaction.rb:308:in `within_new_transaction'                                    
migrations     from /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/connection_adapters/abstract/database_statements.rb:320:in `transaction'                                       
migrations     from /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/transactions.rb:350:in `with_transaction_returning_status'                                                     
migrations     from /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/transactions.rb:298:in `save'                                                                                  
migrations     from /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/suppressor.rb:44:in `save'                                                                                     
migrations     from /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/persistence.rb:617:in `update_attribute'                                                                       
migrations     from /scripts/custom-instance-setup:8:in `<main>'                                                                                                                                                    
migrations     from /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.4.6/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:55:in `load'                                                                   
migrations     from /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.4.6/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:55:in `load'                                                                   
migrations     from /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/railties-6.1.3.2/lib/rails/commands/runner/runner_command.rb:42:in `perform'                                                                          
migrations     from /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/thor-1.1.0/lib/thor/command.rb:27:in `run'