Self hosted CE instance restoring from backup

So I’ve been through the ringer three ways from sunday on this, and I cannot get it to obey me, no matter what I do. Running a self-hosted Gitlab CE instance with an external database. Right now, the gitlab VM is hosted on my infrastructure, and the database and replica servers are hosted externally. I’m trying to make a test setup which includes another gitlab server with external database and replica, but all on my own infrastructure. No matter what I do, the backup will not restore. After many database migrations, I’ve got it narrowed down to “default_branch_name” not being recognized, but it won’t let me into the console to manually fix it. Here’s the error :


root@gitlabtest:/opt/gitlab/embedded/service/gitlab-rails/config# sudo gitlab-rake db:drop db:create db:migrate
sudo gitlab-rake db:setup
sudo gitlab-rake db:seed_fu
rake aborted!
NoMethodError: undefined method `default_branch_name' for #<ApplicationSetting id: 1, site_title: nil, default_timezone: nil, allow_registration: true, maximum_users: 1000, created_at: nil, updated_at: nil, whats_new_variant: nil, email_confirmation_setting: nil, restricted_visibility_levels: nil, import_sources: nil, disabled_oauth_sign_in_sources: nil, domain_allowlist: nil, domain_denylist: nil, asset_proxy_allowlist: nil, asset_proxy_whitelist: nil, repository_storages_weighted: {}, kroki_formats: {}, default_branch_protection_defaults: {}, sidekiq_job_limiter_mode: nil, asset_proxy_secret_key: nil, external_auth_client_key: nil, external_auth_client_key_pass: nil, lets_encrypt_private_key: nil, eks_secret_access_key: nil, akismet_api_key: nil, spam_check_api_key: nil, elasticsearch_aws_secret_access_key: nil, elasticsearch_password: nil, recaptcha_private_key: nil, recaptcha_site_key: nil, slack_app_secret: nil, slack_app_signing_secret: nil, slack_app_verification_token: nil, ci_jwt_signing_key: nil, customers_dot_jwt_signing_key: nil, secret_detection_token_revocation_token: nil, cloud_license_auth_token: nil, external_pipeline_validation_service_token: nil, mailgun_signing_key: nil, database_grafana_api_key: nil, arkose_labs_public_api_key: nil, arkose_labs_private_api_key: nil, arkose_labs_data_exchange_key: nil, cube_api_key: nil, telesign_customer_xid: nil, telesign_api_key: nil, product_analytics_configurator_connection_string: nil, openai_api_key: nil, anthropic_api_key: nil, vertex_ai_credentials: nil, vertex_ai_access_token: nil>
Did you mean?  default_branch_protection_defaults
/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/sanitizable.rb:36:in `public_send'
/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/sanitizable.rb:36:in `block (3 levels) in sanitizes!'
/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/sanitizable.rb:35:in `each'
/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/sanitizable.rb:35:in `block (2 levels) in sanitizes!'
<internal:kernel>:90:in `tap'
/opt/gitlab/embedded/service/gitlab-rails/app/models/application_setting_implementation.rb:307:in `create_from_defaults'
/opt/gitlab/embedded/service/gitlab-rails/app/models/application_setting.rb:898:in `block in create_from_defaults'
/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/cross_database_modification.rb:92:in `block in transaction'
/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/cross_database_modification.rb:83:in `transaction'
/opt/gitlab/embedded/service/gitlab-rails/app/models/application_setting.rb:897:in `create_from_defaults'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/current_settings.rb:71:in `uncached_application_settings'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/current_settings.rb:42:in `ensure_application_settings!'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/current_settings.rb:15:in `block in current_application_settings'
/opt/gitlab/embedded/service/gitlab-rails/gems/gitlab-safe_request_store/lib/gitlab/safe_request_store/null_store.rb:37:in `fetch'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/current_settings.rb:15:in `current_application_settings'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/current_settings.rb:32:in `method_missing'
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/8_devise.rb:165:in `block in <top (required)>'
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/8_devise.rb:7:in `<top (required)>'
/opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:7:in `<top (required)>'
<internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:38:in `require'
<internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:38:in `require'
/opt/gitlab/embedded/bin/bundle:25:in `load'
/opt/gitlab/embedded/bin/bundle:25:in `<main>'
Tasks: TOP => db:drop => db:load_config => environment
(See full trace by running task with --trace)
rake aborted!
NoMethodError: undefined method `default_branch_name' for #<ApplicationSetting id: 1, site_title: nil, default_timezone: nil, allow_registration: true, maximum_users: 1000, created_at: nil, updated_at: nil, whats_new_variant: nil, email_confirmation_setting: nil, restricted_visibility_levels: nil, import_sources: nil, disabled_oauth_sign_in_sources: nil, domain_allowlist: nil, domain_denylist: nil, asset_proxy_allowlist: nil, asset_proxy_whitelist: nil, repository_storages_weighted: {}, kroki_formats: {}, default_branch_protection_defaults: {}, sidekiq_job_limiter_mode: nil, asset_proxy_secret_key: nil, external_auth_client_key: nil, external_auth_client_key_pass: nil, lets_encrypt_private_key: nil, eks_secret_access_key: nil, akismet_api_key: nil, spam_check_api_key: nil, elasticsearch_aws_secret_access_key: nil, elasticsearch_password: nil, recaptcha_private_key: nil, recaptcha_site_key: nil, slack_app_secret: nil, slack_app_signing_secret: nil, slack_app_verification_token: nil, ci_jwt_signing_key: nil, customers_dot_jwt_signing_key: nil, secret_detection_token_revocation_token: nil, cloud_license_auth_token: nil, external_pipeline_validation_service_token: nil, mailgun_signing_key: nil, database_grafana_api_key: nil, arkose_labs_public_api_key: nil, arkose_labs_private_api_key: nil, arkose_labs_data_exchange_key: nil, cube_api_key: nil, telesign_customer_xid: nil, telesign_api_key: nil, product_analytics_configurator_connection_string: nil, openai_api_key: nil, anthropic_api_key: nil, vertex_ai_credentials: nil, vertex_ai_access_token: nil>
Did you mean?  default_branch_protection_defaults
/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/sanitizable.rb:36:in `public_send'
/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/sanitizable.rb:36:in `block (3 levels) in sanitizes!'
/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/sanitizable.rb:35:in `each'
/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/sanitizable.rb:35:in `block (2 levels) in sanitizes!'
<internal:kernel>:90:in `tap'
/opt/gitlab/embedded/service/gitlab-rails/app/models/application_setting_implementation.rb:307:in `create_from_defaults'
/opt/gitlab/embedded/service/gitlab-rails/app/models/application_setting.rb:898:in `block in create_from_defaults'
/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/cross_database_modification.rb:92:in `block in transaction'
/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/cross_database_modification.rb:83:in `transaction'
/opt/gitlab/embedded/service/gitlab-rails/app/models/application_setting.rb:897:in `create_from_defaults'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/current_settings.rb:71:in `uncached_application_settings'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/current_settings.rb:42:in `ensure_application_settings!'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/current_settings.rb:15:in `block in current_application_settings'
/opt/gitlab/embedded/service/gitlab-rails/gems/gitlab-safe_request_store/lib/gitlab/safe_request_store/null_store.rb:37:in `fetch'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/current_settings.rb:15:in `current_application_settings'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/current_settings.rb:32:in `method_missing'
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/8_devise.rb:165:in `block in <top (required)>'
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/8_devise.rb:7:in `<top (required)>'
/opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:7:in `<top (required)>'
<internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:38:in `require'
<internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:38:in `require'
/opt/gitlab/embedded/bin/bundle:25:in `load'
/opt/gitlab/embedded/bin/bundle:25:in `<main>'
Tasks: TOP => db:setup => db:create => db:load_config => environment
(See full trace by running task with --trace)
rake aborted!
NoMethodError: undefined method `default_branch_name' for #<ApplicationSetting id: 1, site_title: nil, default_timezone: nil, allow_registration: true, maximum_users: 1000, created_at: nil, updated_at: nil, whats_new_variant: nil, email_confirmation_setting: nil, restricted_visibility_levels: nil, import_sources: nil, disabled_oauth_sign_in_sources: nil, domain_allowlist: nil, domain_denylist: nil, asset_proxy_allowlist: nil, asset_proxy_whitelist: nil, repository_storages_weighted: {}, kroki_formats: {}, default_branch_protection_defaults: {}, sidekiq_job_limiter_mode: nil, asset_proxy_secret_key: nil, external_auth_client_key: nil, external_auth_client_key_pass: nil, lets_encrypt_private_key: nil, eks_secret_access_key: nil, akismet_api_key: nil, spam_check_api_key: nil, elasticsearch_aws_secret_access_key: nil, elasticsearch_password: nil, recaptcha_private_key: nil, recaptcha_site_key: nil, slack_app_secret: nil, slack_app_signing_secret: nil, slack_app_verification_token: nil, ci_jwt_signing_key: nil, customers_dot_jwt_signing_key: nil, secret_detection_token_revocation_token: nil, cloud_license_auth_token: nil, external_pipeline_validation_service_token: nil, mailgun_signing_key: nil, database_grafana_api_key: nil, arkose_labs_public_api_key: nil, arkose_labs_private_api_key: nil, arkose_labs_data_exchange_key: nil, cube_api_key: nil, telesign_customer_xid: nil, telesign_api_key: nil, product_analytics_configurator_connection_string: nil, openai_api_key: nil, anthropic_api_key: nil, vertex_ai_credentials: nil, vertex_ai_access_token: nil>
Did you mean?  default_branch_protection_defaults
/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/sanitizable.rb:36:in `public_send'
/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/sanitizable.rb:36:in `block (3 levels) in sanitizes!'
/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/sanitizable.rb:35:in `each'
/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/sanitizable.rb:35:in `block (2 levels) in sanitizes!'
<internal:kernel>:90:in `tap'
/opt/gitlab/embedded/service/gitlab-rails/app/models/application_setting_implementation.rb:307:in `create_from_defaults'
/opt/gitlab/embedded/service/gitlab-rails/app/models/application_setting.rb:898:in `block in create_from_defaults'
/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/cross_database_modification.rb:92:in `block in transaction'
/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/cross_database_modification.rb:83:in `transaction'
/opt/gitlab/embedded/service/gitlab-rails/app/models/application_setting.rb:897:in `create_from_defaults'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/current_settings.rb:71:in `uncached_application_settings'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/current_settings.rb:42:in `ensure_application_settings!'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/current_settings.rb:15:in `block in current_application_settings'
/opt/gitlab/embedded/service/gitlab-rails/gems/gitlab-safe_request_store/lib/gitlab/safe_request_store/null_store.rb:37:in `fetch'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/current_settings.rb:15:in `current_application_settings'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/current_settings.rb:32:in `method_missing'
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/8_devise.rb:165:in `block in <top (required)>'
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/8_devise.rb:7:in `<top (required)>'
/opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:7:in `<top (required)>'
<internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:38:in `require'
<internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:38:in `require'
/opt/gitlab/embedded/bin/bundle:25:in `load'
/opt/gitlab/embedded/bin/bundle:25:in `<main>'
Tasks: TOP => db:seed_fu => environment
(See full trace by running task with --trace)

Any ideas would be appreciated…