Postgres error on restore

When we ran the Restore (which was running fine) until it fails with the error below , we cannot figure it out

  • Automation-RnD/techie-town/landroval-stub … [DONE]
  • Automation-RnD/flash-cookbooks/gitlab-chopra … [DONE]
  • Automation-RnD/flash-cookbooks/gitlab-runner-register-chef … [DONE]
  • Automation-RnD/flash-cookbooks/gitlab-runner-register … [DONE]
  • genericswitch/Base24EpsDiners … [DONE]
  • genericswitch/Base24epsSBSA … [DONE]
  • A227894/PProject … [DONE]
  • earning-your-wings/wings-testing … [DONE]
  • earning-your-wings/reference-material … rake aborted!
    ActiveRecord::StatementInvalid: PG::SyntaxError: ERROR: zero-length delimited identifier at or near “”""
    LINE 1: …cation_settings" ORDER BY “application_settings”."" DESC LI…
    ^
    : SELECT “application_settings”.* FROM “application_settings” ORDER BY “application_settings”."" DESC LIMIT 1
    /opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/cacheable_attributes.rb:17:in current_without_cache' /opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/cacheable_attributes.rb:52:in rescue in current’
    /opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/cacheable_attributes.rb:40:in current' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/current_settings.rb:45:in uncached_application_settings’
    /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/current_settings.rb:27:in ensure_application_settings!' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/current_settings.rb:5:in block in current_application_settings’
    /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/null_request_store.rb:34:in fetch' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/safe_request_store.rb:12:in fetch’
    /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/current_settings.rb:5:in current_application_settings' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client.rb:442:in timeout’
    /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client.rb:430:in fast_timeout' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client/namespace_service.rb:17:in add’
    /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/shell.rb:282:in add_namespace' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/instrumentation.rb:161:in add_namespace’
    /opt/gitlab/embedded/service/gitlab-rails/app/models/storage/legacy_project.rb:29:in ensure_storage_path_exists' /opt/gitlab/embedded/service/gitlab-rails/app/models/project.rb:61:in ensure_storage_path_exists’
    /opt/gitlab/embedded/service/gitlab-rails/lib/backup/repository.rb:83:in block in restore' /opt/gitlab/embedded/service/gitlab-rails/lib/backup/repository.rb:80:in restore’
    /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:87:in block (4 levels) in <top (required)>' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:60:in block (3 levels) in <top (required)>’
    /opt/gitlab/embedded/bin/bundle:23:in load' /opt/gitlab/embedded/bin/bundle:23:in

Caused by:
PG::SyntaxError: ERROR: zero-length delimited identifier at or near “”""
LINE 1: …cation_settings" ORDER BY “application_settings”."" DESC LI…
^
/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/cacheable_attributes.rb:17:in current_without_cache' /opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/cacheable_attributes.rb:52:in rescue in current’
/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/cacheable_attributes.rb:40:in current' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/current_settings.rb:45:in uncached_application_settings’
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/current_settings.rb:27:in ensure_application_settings!' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/current_settings.rb:5:in block in current_application_settings’
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/null_request_store.rb:34:in fetch' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/safe_request_store.rb:12:in fetch’
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/current_settings.rb:5:in current_application_settings' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client.rb:442:in timeout’
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client.rb:430:in fast_timeout' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client/namespace_service.rb:17:in add’
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/shell.rb:282:in add_namespace' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/instrumentation.rb:161:in add_namespace’
/opt/gitlab/embedded/service/gitlab-rails/app/models/storage/legacy_project.rb:29:in ensure_storage_path_exists' /opt/gitlab/embedded/service/gitlab-rails/app/models/project.rb:61:in ensure_storage_path_exists’
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/repository.rb:83:in block in restore' /opt/gitlab/embedded/service/gitlab-rails/lib/backup/repository.rb:80:in restore’
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:87:in block (4 levels) in <top (required)>' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:60:in block (3 levels) in <top (required)>’
/opt/gitlab/embedded/bin/bundle:23:in load' /opt/gitlab/embedded/bin/bundle:23:in

Caused by:
ActiveRecord::StatementInvalid: PG::SyntaxError: ERROR: zero-length delimited identifier at or near “”""
LINE 1: …cation_settings" ORDER BY “application_settings”."" DESC LI…
^
: SELECT “application_settings”.* FROM “application_settings” ORDER BY “application_settings”."" DESC LIMIT 1
/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/cacheable_attributes.rb:17:in current_without_cache' /opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/cacheable_attributes.rb:44:in current’
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/current_settings.rb:45:in uncached_application_settings' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/current_settings.rb:27:in ensure_application_settings!’
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/current_settings.rb:5:in block in current_application_settings' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/null_request_store.rb:34:in fetch’
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/safe_request_store.rb:12:in fetch' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/current_settings.rb:5:in current_application_settings’
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client.rb:442:in timeout' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client.rb:430:in fast_timeout’
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client/namespace_service.rb:17:in add' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/shell.rb:282:in add_namespace’
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/instrumentation.rb:161:in add_namespace' /opt/gitlab/embedded/service/gitlab-rails/app/models/storage/legacy_project.rb:29:in ensure_storage_path_exists’
/opt/gitlab/embedded/service/gitlab-rails/app/models/project.rb:61:in ensure_storage_path_exists' /opt/gitlab/embedded/service/gitlab-rails/lib/backup/repository.rb:83:in block in restore’
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/repository.rb:80:in restore' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:87:in block (4 levels) in <top (required)>’
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:60:in block (3 levels) in <top (required)>' /opt/gitlab/embedded/bin/bundle:23:in load’
/opt/gitlab/embedded/bin/bundle:23:in `’

We tried running the fix below (Unable to run any gitlab-rake commands and 502 message (#31976) · Issues · GitLab.org / GitLab FOSS · GitLab)

ALTER TABLE ONLY application_settings
ADD CONSTRAINT application_settings_pkey PRIMARY KEY (id);

Results

gitlabhq_production=> ALTER TABLE ONLY application_settings ADD CONSTRAINT application_settings_pkey PRIMARY KEY (id);
ERROR: multiple primary keys for table “application_settings” are not allowed