Database errors during backup restore

Hi!

I am migrating my gitlab from one server to another. My steps:

1. Update old gitlab-ce to latest version (11.0.2, omnibus, ubuntu 16.04)
2. check functionality after upgrade (ok)
3. generate backup
4. copy backup over scp to new server
5. install fresh gitlab-ce (11.0.2, omnibus, ubuntu 18.04)
6. configure new gitlab and check functionality (ok)
7. restore backup

When I restore backup I get ton of errors:

ERROR:  relation "public.timelogs" does not exist
ERROR:  relation "public.timelogs" does not exist
ERROR:  relation "public.ci_builds_metadata" does not exist
ERROR:  relation "public.ci_runner_namespaces" does not exist
ERROR:  relation "public.ci_build_trace_section_names" does not exist
ERROR:  relation "public.merge_requests_closing_issues" does not exist
ERROR:  relation "public.internal_ids" does not exist
ERROR:  relation "public.events" does not exist
ERROR:  relation "public.fork_network_members" does not exist
ERROR:  relation "public.label_priorities" does not exist
ERROR:  relation "public.merge_request_metrics" does not exist
ERROR:  relation "public.ci_builds_metadata" does not exist
ERROR:  relation "public.cluster_platforms_kubernetes" does not exist
ERROR:  relation "public.label_priorities" does not exist
ERROR:  relation "public.user_callouts" does not exist
ERROR:  relation "public.gcp_clusters" does not exist
ERROR:  relation "public.project_mirror_data" does not exist
ERROR:  relation "public.subscriptions" does not exist
ERROR:  relation "public.gpg_signatures" does not exist
ERROR:  relation "public.ci_job_artifacts" does not exist
ERROR:  relation "public.labels" does not exist
ERROR:  relation "public.u2f_registrations" does not exist
ERROR:  relation "public.protected_tag_create_access_levels" does not exist
ERROR:  relation "public.gcp_clusters" does not exist
ERROR:  relation "public.clusters" does not exist
ERROR:  relation "public.ci_build_trace_sections" does not exist
ERROR:  relation "public.term_agreements" does not exist
ERROR:  relation "public.cluster_projects" does not exist
ERROR:  relation "public.fork_network_members" does not exist
ERROR:  relation "public.badges" does not exist
ERROR:  relation "public.gpg_keys" does not exist
ERROR:  relation "public.ci_job_artifacts" does not exist
ERROR:  relation "public.cluster_projects" does not exist
ERROR:  relation "public.gpg_key_subkeys" does not exist
ERROR:  relation "public.ci_runner_namespaces" does not exist
ERROR:  relation "public.clusters_applications_ingress" does not exist
ERROR:  relation "public.user_interacted_projects" does not exist
ERROR:  relation "public.project_custom_attributes" does not exist
ERROR:  relation "public.term_agreements" does not exist
ERROR:  relation "public.web_hook_logs" does not exist
ERROR:  relation "public.cluster_providers_gcp" does not exist
ERROR:  relation "public.badges" does not exist
ERROR:  relation "public.clusters_applications_prometheus" does not exist
ERROR:  relation "public.merge_request_diff_files" does not exist
ERROR:  relation "public.issue_metrics" does not exist
ERROR:  relation "public.user_custom_attributes" does not exist
ERROR:  relation "public.merge_requests_closing_issues" does not exist
ERROR:  relation "public.project_auto_devops" does not exist
ERROR:  relation "public.lfs_file_locks" does not exist
ERROR:  relation "public.remote_mirrors" does not exist
ERROR:  relation "public.gcp_clusters" does not exist
ERROR:  relation "public.clusters_applications_helm" does not exist
ERROR:  relation "public.note_diff_files" does not exist
ERROR:  relation "public.chat_teams" does not exist
ERROR:  relation "public.merge_request_metrics" does not exist
ERROR:  relation "public.clusters_applications_jupyter" does not exist
ERROR:  relation "public.container_repositories" does not exist
ERROR:  relation "public.merge_request_diff_commits" does not exist
ERROR:  relation "public.lfs_file_locks" does not exist
ERROR:  relation "public.group_custom_attributes" does not exist
ERROR:  relation "public.protected_tag_create_access_levels" does not exist
ERROR:  relation "public.clusters_applications_runners" does not exist
ERROR:  relation "public.boards" does not exist
ERROR:  relation "public.gpg_signatures" does not exist
ERROR:  relation "public.clusters_applications_jupyter" does not exist
ERROR:  relation "public.project_deploy_tokens" does not exist
ERROR:  relation "public.project_statistics" does not exist
ERROR:  relation "public.project_authorizations" does not exist
ERROR:  relation "public.gpg_signatures" does not exist
ERROR:  relation "public.ci_build_trace_chunks" does not exist
ERROR:  relation "public.project_authorizations" does not exist
ERROR:  relation "public.user_synced_attributes_metadata" does not exist
ERROR:  relation "public.project_deploy_tokens" does not exist
ERROR:  relation "public.trending_projects" does not exist
ERROR:  relation "public.user_interacted_projects" does not exist
ERROR:  relation "public.personal_access_tokens" does not exist
ERROR:  relation "public.oauth_openid_requests" does not exist
ERROR:  relation "public.issues" does not exist
ERROR:  relation "public.project_import_data" does not exist
ERROR:  relation "public.merge_requests" does not exist
ERROR:  relation "public.ci_stages" does not exist
ERROR:  relation "public.protected_tag_create_access_levels" does not exist
ERROR:  relation "public.ci_pipeline_variables" does not exist
ERROR:  relation "public.boards" does not exist
ERROR:  relation "public.events" does not exist
ERROR:  relation "public.pages_domains" does not exist
ERROR:  relation "public.ci_triggers" does not exist
ERROR:  relation "public.fork_networks" does not exist
ERROR:  relation "public.merge_requests" does not exist
ERROR:  relation "public.ci_triggers" does not exist
ERROR:  relation "public.project_group_links" does not exist
ERROR:  relation "public.todos" does not exist
ERROR:  relation "public.system_note_metadata" does not exist
ERROR:  relation "public.ci_builds" does not exist
ERROR:  relation "public.environments" does not exist
ERROR:  relation "public.todos" does not exist
ERROR:  relation "public.issues" does not exist
ERROR:  relation "public.ci_builds" does not exist
ERROR:  relation "public.snippets" does not exist
ERROR:  relation "public.deployments" does not exist
ERROR:  relation "public.ci_trigger_requests" does not exist
ERROR:  relation "public.issue_assignees" does not exist
ERROR:  relation "public.fork_network_members" does not exist
ERROR:  relation "public.merge_request_metrics" does not exist
ERROR:  relation "public.ci_variables" does not exist
ERROR:  relation "public.merge_requests" does not exist
ERROR:  relation "public.merge_requests" does not exist
ERROR:  relation "public.ci_builds" does not exist
ERROR:  relation "public.issues" does not exist
ERROR:  relation "public.protected_branch_push_access_levels" does not exist
ERROR:  relation "public.ci_pipeline_schedules" does not exist
ERROR:  relation "public.milestones" does not exist
ERROR:  relation "public.notes" does not exist
ERROR:  relation "public.issues" does not exist
ERROR:  relation "public.milestones" does not exist
ERROR:  relation "public.todos" does not exist
ERROR:  relation "public.ci_pipeline_schedules" does not exist
ERROR:  relation "public.protected_tags" does not exist
ERROR:  relation "public.protected_branch_merge_access_levels" does not exist
ERROR:  relation "public.issues" does not exist
ERROR:  relation "public.ci_pipelines" does not exist
ERROR:  relation "public.merge_request_diffs" does not exist
ERROR:  relation "public.merge_request_metrics" does not exist
ERROR:  relation "public.labels" does not exist
ERROR:  relation "public.protected_branches" does not exist
ERROR:  relation "public.lists" does not exist
ERROR:  relation "public.users" does not exist
ERROR:  relation "public.services" does not exist
ERROR:  relation "public.merge_requests" does not exist
ERROR:  relation "public.merge_requests" does not exist
ERROR:  relation "public.merge_requests" does not exist
ERROR:  relation "public.issue_assignees" does not exist
ERROR:  relation "public.deploy_keys_projects" does not exist
ERROR:  relation "public.ci_build_trace_sections" does not exist
ERROR:  relation "public.releases" does not exist
ERROR:  relation "public.todos" does not exist
ERROR:  relation "public.ci_runner_projects" does not exist
ERROR:  relation "public.forked_project_links" does not exist
ERROR:  relation "public.ci_pipeline_schedule_variables" does not exist
ERROR:  relation "public.ci_pipelines" does not exist
ERROR:  relation "public.ci_builds" does not exist
ERROR:  relation "public.push_event_payloads" does not exist
ERROR:  relation "public.ci_group_variables" does not exist
ERROR:  relation "public.merge_requests" does not exist
ERROR:  relation "public.members" does not exist
ERROR:  relation "public.ci_build_trace_sections" does not exist
ERROR:  relation "public.ci_pipelines" does not exist
ERROR:  relation "public.project_ci_cd_settings" does not exist
ERROR:  relation "public.ci_stages" does not exist
ERROR:  relation "public.users_star_projects" does not exist
ERROR:  relation "public.project_features" does not exist
ERROR:  relation "public.internal_ids" does not exist
ERROR:  relation "public.lists" does not exist
ERROR:  relation "public.web_hooks" does not exist
ERROR:  relation "public.merge_requests" does not exist
ERROR:  relation "public.issues" does not exist
ERROR:  relation "public.clusters_applications_runners" does not exist
DROP INDEX
ERROR:  index "tmp_build_stage_position_index" does not exist
ERROR:  index "term_agreements_unique_index" does not exist
ERROR:  index "taggings_idx" does not exist
ERROR:  index "index_web_hooks_on_type" does not exist
ERROR:  index "index_web_hooks_on_project_id" does not exist
ERROR:  index "index_web_hook_logs_on_web_hook_id" does not exist
ERROR:  index "index_users_star_projects_on_user_id_and_project_id" does not exist
ERROR:  index "index_users_star_projects_on_project_id" does not exist
ERROR:  index "index_users_on_username_trigram" does not exist
ERROR:  index "index_users_on_username" does not exist
ERROR:  index "index_users_on_state" does not exist
ERROR:  index "index_users_on_reset_password_token" does not exist
ERROR:  index "index_users_on_name_trigram" does not exist
ERROR:  index "index_users_on_name" does not exist
ERROR:  index "index_users_on_incoming_email_token" does not exist
ERROR:  index "index_users_on_ghost" does not exist
ERROR:  index "index_users_on_feed_token" does not exist
ERROR:  index "index_users_on_email_trigram" does not exist
ERROR:  index "index_users_on_email" does not exist
ERROR:  index "index_users_on_created_at" does not exist
ERROR:  index "index_users_on_confirmation_token" does not exist
ERROR:  index "index_users_on_admin" does not exist
ERROR:  index "index_user_synced_attributes_metadata_on_user_id" does not exist
ERROR:  index "index_user_interacted_projects_on_user_id" does not exist
ERROR:  index "index_user_interacted_projects_on_project_id_and_user_id" does not exist
ERROR:  index "index_user_custom_attributes_on_user_id_and_key" does not exist
ERROR:  index "index_user_custom_attributes_on_key_and_value" does not exist
ERROR:  index "index_user_callouts_on_user_id_and_feature_name" does not exist
ERROR:  index "index_user_callouts_on_user_id" does not exist
ERROR:  index "index_user_agent_details_on_subject_id_and_subject_type" does not exist
ERROR:  index "index_uploads_on_uploader_and_path" does not exist
ERROR:  index "index_uploads_on_model_id_and_model_type" does not exist
ERROR:  index "index_uploads_on_checksum" does not exist
ERROR:  index "index_u2f_registrations_on_user_id" does not exist
ERROR:  index "index_u2f_registrations_on_key_handle" does not exist
ERROR:  index "index_trending_projects_on_project_id" does not exist
ERROR:  index "index_todos_on_user_id_and_id_pending" does not exist
ERROR:  index "index_todos_on_user_id_and_id_done" does not exist
ERROR:  index "index_todos_on_user_id" does not exist
ERROR:  index "index_todos_on_target_type_and_target_id" does not exist
ERROR:  index "index_todos_on_project_id" does not exist
ERROR:  index "index_todos_on_note_id" does not exist
ERROR:  index "index_todos_on_commit_id" does not exist
ERROR:  index "index_todos_on_author_id" does not exist
ERROR:  index "index_timelogs_on_user_id" does not exist
ERROR:  index "index_timelogs_on_merge_request_id" does not exist
ERROR:  index "index_timelogs_on_issue_id" does not exist
ERROR:  index "index_term_agreements_on_user_id" does not exist
ERROR:  index "index_term_agreements_on_term_id" does not exist
ERROR:  index "index_tags_on_name" does not exist
ERROR:  index "index_taggings_on_taggable_id_and_taggable_type_and_context" does not exist
ERROR:  index "index_taggings_on_taggable_id_and_taggable_type" does not exist
ERROR:  index "index_taggings_on_tag_id" does not exist
ERROR:  index "index_system_note_metadata_on_note_id" does not exist
ERROR:  index "index_subscriptions_on_subscribable_and_user_id_and_project_id" does not exist
ERROR:  index "index_snippets_on_visibility_level" does not exist
ERROR:  index "index_snippets_on_updated_at" does not exist
ERROR:  index "index_snippets_on_title_trigram" does not exist
ERROR:  index "index_snippets_on_project_id" does not exist
ERROR:  index "index_snippets_on_file_name_trigram" does not exist
ERROR:  index "index_snippets_on_author_id" does not exist
ERROR:  index "index_services_on_template" does not exist
ERROR:  index "index_services_on_project_id" does not exist
ERROR:  index "index_sent_notifications_on_reply_key" does not exist
ERROR:  index "index_routes_on_source_type_and_source_id" does not exist
ERROR:  index "index_routes_on_path_text_pattern_ops" does not exist
ERROR:  index "index_routes_on_path" does not exist
ERROR:  index "index_remote_mirrors_on_project_id" does not exist
ERROR:  index "index_remote_mirrors_on_last_successful_update_at" does not exist
ERROR:  index "index_releases_on_project_id_and_tag" does not exist
ERROR:  index "index_releases_on_project_id" does not exist

(truncated - a lot of similar errors happened here)

ERROR:  table "ci_stages" does not exist
ERROR:  sequence "ci_runners_id_seq" does not exist
ERROR:  table "ci_runners" does not exist
ERROR:  sequence "ci_runner_projects_id_seq" does not exist
ERROR:  table "ci_runner_projects" does not exist
ERROR:  sequence "ci_runner_namespaces_id_seq" does not exist
ERROR:  table "ci_runner_namespaces" does not exist
ERROR:  sequence "ci_pipelines_id_seq" does not exist
ERROR:  table "ci_pipelines" does not exist
ERROR:  sequence "ci_pipeline_variables_id_seq" does not exist
ERROR:  table "ci_pipeline_variables" does not exist
ERROR:  sequence "ci_pipeline_schedules_id_seq" does not exist
ERROR:  table "ci_pipeline_schedules" does not exist
ERROR:  sequence "ci_pipeline_schedule_variables_id_seq" does not exist
ERROR:  table "ci_pipeline_schedule_variables" does not exist
ERROR:  sequence "ci_job_artifacts_id_seq" does not exist
ERROR:  table "ci_job_artifacts" does not exist
ERROR:  sequence "ci_group_variables_id_seq" does not exist
ERROR:  table "ci_group_variables" does not exist
ERROR:  sequence "ci_builds_metadata_id_seq" does not exist
ERROR:  table "ci_builds_metadata" does not exist
ERROR:  sequence "ci_builds_id_seq" does not exist
ERROR:  table "ci_builds" does not exist
ERROR:  sequence "ci_build_trace_sections_id_seq" does not exist
ERROR:  table "ci_build_trace_sections" does not exist
ERROR:  sequence "ci_build_trace_section_names_id_seq" does not exist
ERROR:  table "ci_build_trace_section_names" does not exist
ERROR:  sequence "ci_build_trace_chunks_id_seq" does not exist
ERROR:  table "ci_build_trace_chunks" does not exist
ERROR:  sequence "chat_teams_id_seq" does not exist
ERROR:  table "chat_teams" does not exist
ERROR:  sequence "chat_names_id_seq" does not exist
ERROR:  table "chat_names" does not exist
ERROR:  sequence "broadcast_messages_id_seq" does not exist
ERROR:  table "broadcast_messages" does not exist
ERROR:  sequence "boards_id_seq" does not exist
ERROR:  table "boards" does not exist
ERROR:  sequence "badges_id_seq" does not exist
ERROR:  table "badges" does not exist
ERROR:  sequence "award_emoji_id_seq" does not exist
ERROR:  table "award_emoji" does not exist
ERROR:  sequence "audit_events_id_seq" does not exist
ERROR:  table "audit_events" does not exist
ERROR:  sequence "application_settings_id_seq" does not exist
ERROR:  table "application_settings" does not exist
ERROR:  sequence "application_setting_terms_id_seq" does not exist
ERROR:  table "application_setting_terms" does not exist
ERROR:  sequence "appearances_id_seq" does not exist
ERROR:  table "appearances" does not exist
ERROR:  sequence "abuse_reports_id_seq" does not exist
ERROR:  table "abuse_reports" does not exist
ERROR:  must be owner of extension pg_trgm
ERROR:  must be owner of extension plpgsql
ERROR:  must be owner of schema public
ERROR:  schema "public" already exists
ALTER SCHEMA
ERROR:  must be owner of schema public
CREATE EXTENSION
ERROR:  must be owner of extension plpgsql
CREATE EXTENSION
ERROR:  must be owner of extension pg_trgm

After restoring .secrets file and running sudo gitlab-rake gitlab:check SANITIZE=true gitlab seems to run fine.

Are those errors expected? Shoud I be worried?

Hi.

When you restore the backup, it asks you a question "removing all existing
tables …

Do you want to continue (yes / no)? "

Did you write yes?

If you wrote no, try with yes.

Hi,

I chose “yes”, the database should have been cleared.

Hi all,

I am seeing pretty much exactly the same behavior as the original poster in several versions of Gitlab including the current version 12.2.1. In order to exclude as many possible sources as possible I have set up a fresh gitlab installation (using the CE docker image) and did a backup/restore without modifying anything.

Following the docu, I

• stopped unicorn and sidekiq
• did the restore (confirming the removal of existing tables)
• restarted gitlab
• after a couple of minutes checked using sudo docker exec gitlab gitlab-rake gitlab:check SANITIZE=true

I am seeing quite exactly the same errors as the original poster. Despite the error messages, the restore finishes and the check does not report any problems. Naturally, I am also asking myself the same questions:

• Is this expected?
• Do I need to worry?

If this is expected how do you detect a real problem?

Thanks for your help and best wishes,
Lutz