Gitlab:backup:restore freezes while dropping database indexes

Dear Community,
I have a problem restoring a backup.
For showing of this roblem I use follow steps:

  1. gitlab/gitlab-ce:11.3.0-ce.0 Image on my docker host with default Parameters
  2. gitlab-rake gitlab:backup:create for create backup
  3. Stopping all gitlab services but postgresql
  4. gitlab-rake gitlab:backup:restore And it frozen now!

I attach here the log file of gitlab-rake gitlab:backup:restore. And I hope very much, that some one of You have can help me.

Thank You very much!
JeMo

root@gitlab-2-86c86d4cdb-8rlbs:/# gitlab-rake gitlab:backup:restore
Unpacking backup … done
Before restoring the database, we will remove all existing
tables to avoid future upgrade problems. Be aware that if you have
custom tables in the GitLab database these tables and all data will be
removed.

Do you want to continue (yes/no)? yes
Removing all tables. Press Ctrl-C within 5 seconds to abort
Cleaning the database …
done
Restoring database …
Restoring PostgreSQL database gitlabhq_production … SET
SET
SET
SET
SET
set_config

(1 row)

SET
SET
SET
ERROR: relation “public.timelogs” does not exist
ERROR: relation “public.timelogs” does not exist
ERROR: relation “public.ci_builds_metadata” does not exist
ERROR: relation “public.resource_label_events” does not exist
ERROR: relation “public.ci_runner_namespaces” does not exist
ERROR: relation “public.ci_build_trace_section_names” does not exist
ERROR: relation “public.merge_requests_closing_issues” does not exist
ERROR: relation “public.internal_ids” does not exist
ERROR: relation “public.import_export_uploads” does not exist
ERROR: relation “public.fork_network_members” does not exist
ERROR: relation “public.label_priorities” does not exist
ERROR: relation “public.merge_request_metrics” does not exist
ERROR: relation “public.ci_builds_metadata” does not exist
ERROR: relation “public.cluster_platforms_kubernetes” does not exist
ERROR: relation “public.label_priorities” does not exist
ERROR: relation “public.user_callouts” does not exist
ERROR: relation “public.gcp_clusters” does not exist
ERROR: relation “public.project_mirror_data” does not exist
ERROR: relation “public.subscriptions” does not exist
ERROR: relation “public.gpg_signatures” does not exist
ERROR: relation “public.ci_job_artifacts” does not exist
ERROR: relation “public.labels” does not exist
ERROR: relation “public.u2f_registrations” does not exist
ERROR: relation “public.protected_tag_create_access_levels” does not exist
ERROR: relation “public.gcp_clusters” does not exist
ERROR: relation “public.resource_label_events” does not exist
ERROR: relation “public.clusters” does not exist
ERROR: relation “public.ci_build_trace_sections” does not exist
ERROR: relation “public.term_agreements” does not exist
ERROR: relation “public.repository_languages” does not exist
ERROR: relation “public.cluster_projects” does not exist
ERROR: relation “public.fork_network_members” does not exist
ERROR: relation “public.todos” does not exist
ERROR: relation “public.badges” does not exist
ERROR: relation “public.gpg_keys” does not exist
ERROR: relation “public.ci_job_artifacts” does not exist
ERROR: relation “public.resource_label_events” does not exist
ERROR: relation “public.cluster_projects” does not exist
ERROR: relation “public.gpg_key_subkeys” does not exist
ERROR: relation “public.ci_runner_namespaces” does not exist
ERROR: relation “public.project_custom_attributes” does not exist
ERROR: relation “public.ci_builds_runner_session” does not exist
ERROR: relation “public.term_agreements” does not exist
ERROR: relation “public.web_hook_logs” does not exist
ERROR: relation “public.cluster_providers_gcp” does not exist
ERROR: relation “public.resource_label_events” does not exist
ERROR: relation “public.badges” does not exist
ERROR: relation “public.merge_request_diff_files” does not exist
ERROR: relation “public.prometheus_metrics” does not exist
ERROR: relation “public.issue_metrics” does not exist
ERROR: relation “public.user_custom_attributes” does not exist
ERROR: relation “public.merge_requests_closing_issues” does not exist
ERROR: relation “public.project_auto_devops” does not exist
ERROR: relation “public.lfs_file_locks” does not exist
ERROR: relation “public.remote_mirrors” does not exist
ERROR: relation “public.gcp_clusters” does not exist
ERROR: relation “public.clusters_applications_helm” does not exist
ERROR: relation “public.note_diff_files” does not exist
ERROR: relation “public.chat_teams” does not exist
ERROR: relation “public.merge_request_metrics” does not exist
ERROR: relation “public.clusters_applications_jupyter” does not exist
ERROR: relation “public.container_repositories” does not exist
ERROR: relation “public.merge_request_diff_commits” does not exist
ERROR: relation “public.lfs_file_locks” does not exist
ERROR: relation “public.group_custom_attributes” does not exist
ERROR: relation “public.protected_tag_create_access_levels” does not exist
ERROR: relation “public.clusters_applications_runners” does not exist
ERROR: relation “public.user_statuses” does not exist
ERROR: relation “public.boards” does not exist
ERROR: relation “public.gpg_signatures” does not exist
ERROR: relation “public.clusters_applications_jupyter” does not exist
ERROR: relation “public.project_deploy_tokens” does not exist
ERROR: relation “public.project_statistics” does not exist
ERROR: relation “public.project_authorizations” does not exist
ERROR: relation “public.gpg_signatures” does not exist
ERROR: relation “public.ci_build_trace_chunks” does not exist
ERROR: relation “public.project_authorizations” does not exist
ERROR: relation “public.user_synced_attributes_metadata” does not exist
ERROR: relation “public.project_deploy_tokens” does not exist
ERROR: relation “public.trending_projects” does not exist
ERROR: relation “public.personal_access_tokens” does not exist
ERROR: relation “public.events” does not exist
ERROR: relation “public.oauth_openid_requests” does not exist
ERROR: relation “public.issues” does not exist
ERROR: relation “public.project_import_data” does not exist
ERROR: relation “public.merge_requests” does not exist
ERROR: relation “public.ci_stages” does not exist
ERROR: relation “public.protected_tag_create_access_levels” does not exist
ERROR: relation “public.ci_pipeline_variables” does not exist
ERROR: relation “public.boards” does not exist
ERROR: relation “public.events” does not exist
ERROR: relation “public.pages_domains” does not exist
ERROR: relation “public.ci_triggers” does not exist
ERROR: relation “public.fork_networks” does not exist
ERROR: relation “public.merge_requests” does not exist
ERROR: relation “public.ci_triggers” does not exist
ERROR: relation “public.project_group_links” does not exist
ERROR: relation “public.label_links” does not exist
ERROR: relation “public.todos” does not exist
ERROR: relation “public.system_note_metadata” does not exist
ERROR: relation “public.ci_builds” does not exist
ERROR: relation “public.environments” does not exist
ERROR: relation “public.todos” does not exist
ERROR: relation “public.issues” does not exist
ERROR: relation “public.ci_builds” does not exist
ERROR: relation “public.snippets” does not exist
ERROR: relation “public.deployments” does not exist
ERROR: relation “public.ci_trigger_requests” does not exist
ERROR: relation “public.issue_assignees” does not exist
ERROR: relation “public.fork_network_members” does not exist
ERROR: relation “public.merge_request_metrics” does not exist
ERROR: relation “public.ci_variables” does not exist
ERROR: relation “public.merge_requests” does not exist
ERROR: relation “public.merge_requests” does not exist
ERROR: relation “public.ci_builds” does not exist
ERROR: relation “public.issues” does not exist
ERROR: relation “public.protected_branch_push_access_levels” does not exist
ERROR: relation “public.ci_pipeline_schedules” does not exist
ERROR: relation “public.milestones” does not exist
ERROR: relation “public.notes” does not exist
ERROR: relation “public.issues” does not exist
ERROR: relation “public.application_settings” does not exist
ERROR: relation “public.milestones” does not exist
ERROR: relation “public.todos” does not exist
ERROR: relation “public.ci_pipeline_schedules” does not exist
ERROR: relation “public.protected_tags” does not exist
ERROR: relation “public.protected_branch_merge_access_levels” does not exist
ERROR: relation “public.issues” does not exist
ERROR: relation “public.ci_pipelines” does not exist
ERROR: relation “public.merge_request_diffs” does not exist
ERROR: relation “public.merge_request_metrics” does not exist
ERROR: relation “public.labels” does not exist
ERROR: relation “public.protected_branches” does not exist
ERROR: relation “public.lists” does not exist
ERROR: relation “public.users” does not exist
ERROR: relation “public.clusters_applications_ingress” does not exist
ERROR: relation “public.user_interacted_projects” does not exist
ERROR: relation “public.services” does not exist
ERROR: relation “public.merge_requests” does not exist
ERROR: relation “public.merge_requests” does not exist
ERROR: relation “public.merge_requests” does not exist
ERROR: relation “public.issue_assignees” does not exist
ERROR: relation “public.deploy_keys_projects” does not exist
ERROR: relation “public.clusters_applications_prometheus” does not exist
ERROR: relation “public.ci_build_trace_sections” does not exist
ERROR: relation “public.releases” does not exist
ERROR: relation “public.todos” does not exist
ERROR: relation “public.ci_runner_projects” does not exist
ERROR: relation “public.forked_project_links” does not exist
ERROR: relation “public.ci_pipeline_schedule_variables” does not exist
ERROR: relation “public.ci_pipelines” does not exist
ERROR: relation “public.ci_builds” does not exist
ERROR: relation “public.push_event_payloads” does not exist
ERROR: relation “public.ci_group_variables” does not exist
ERROR: relation “public.merge_requests” does not exist
ERROR: relation “public.members” does not exist
ERROR: relation “public.ci_build_trace_sections” does not exist
ERROR: relation “public.ci_pipelines” does not exist
ERROR: relation “public.project_ci_cd_settings” does not exist
ERROR: relation “public.ci_stages” does not exist
ERROR: relation “public.users_star_projects” does not exist
ERROR: relation “public.project_features” does not exist
ERROR: relation “public.internal_ids” does not exist
ERROR: relation “public.lists” does not exist
ERROR: relation “public.notification_settings” does not exist
ERROR: relation “public.web_hooks” does not exist
ERROR: relation “public.user_interacted_projects” does not exist
ERROR: relation “public.merge_requests” does not exist
ERROR: relation “public.issues” does not exist
ERROR: relation “public.clusters_applications_runners” does not exist
ERROR: index “unique_schema_migrations” does not exist
ERROR: index “tmp_build_stage_position_index” does not exist
ERROR: index “term_agreements_unique_index” does not exist
ERROR: index “taggings_idx” does not exist
ERROR: index “partial_index_ci_builds_on_id_with_legacy_artifacts” does not exist
ERROR: index “index_web_hooks_on_type” does not exist
ERROR: index “index_web_hooks_on_project_id” does not exist
ERROR: index “index_web_hook_logs_on_web_hook_id” does not exist
ERROR: index “index_web_hook_logs_on_created_at_and_web_hook_id” does not exist
ERROR: index “index_users_star_projects_on_user_id_and_project_id” does not exist
ERROR: index “index_users_star_projects_on_project_id” does not exist
ERROR: index “index_users_on_username_trigram” does not exist
ERROR: index “index_users_on_username” does not exist
ERROR: index “index_users_on_state” does not exist
ERROR: index “index_users_on_reset_password_token” does not exist
ERROR: index “index_users_on_name_trigram” does not exist
ERROR: index “index_users_on_name” does not exist
ERROR: index “index_users_on_incoming_email_token” does not exist
ERROR: index “index_users_on_ghost” does not exist
ERROR: index “index_users_on_feed_token” does not exist
ERROR: index “index_users_on_email_trigram” does not exist
ERROR: index “index_users_on_email” does not exist
ERROR: index “index_users_on_created_at” does not exist
ERROR: index “index_users_on_confirmation_token” does not exist
ERROR: index “index_users_on_admin” does not exist
ERROR: index “index_user_synced_attributes_metadata_on_user_id” does not exist
ERROR: index “index_user_interacted_projects_on_user_id” does not exist
ERROR: index “index_user_interacted_projects_on_project_id_and_user_id” does not exist
ERROR: index “index_user_custom_attributes_on_user_id_and_key” does not exist
ERROR: index “index_user_custom_attributes_on_key_and_value” does not exist
ERROR: index “index_user_callouts_on_user_id_and_feature_name” does not exist
ERROR: index “index_user_callouts_on_user_id” does not exist
ERROR: index “index_user_agent_details_on_subject_id_and_subject_type” does not exist
ERROR: index “index_uploads_on_uploader_and_path” does not exist
ERROR: index “index_uploads_on_model_id_and_model_type” does not exist
ERROR: index “index_uploads_on_checksum” does not exist
ERROR: index “index_u2f_registrations_on_user_id” does not exist
ERROR: index “index_u2f_registrations_on_key_handle” does not exist
ERROR: index “index_trending_projects_on_project_id” does not exist
ERROR: index “index_todos_on_user_id_and_id_pending” does not exist
ERROR: index “index_todos_on_user_id_and_id_done” does not exist
ERROR: index “index_todos_on_user_id” does not exist
ERROR: index “index_todos_on_target_type_and_target_id” does not exist
ERROR: index “index_todos_on_project_id” does not exist
ERROR: index “index_todos_on_note_id” does not exist
ERROR: index “index_todos_on_group_id” does not exist
ERROR: index “index_todos_on_commit_id” does not exist
ERROR: index “index_todos_on_author_id” does not exist
ERROR: index “index_timelogs_on_user_id” does not exist
ERROR: index “index_timelogs_on_merge_request_id” does not exist
ERROR: index “index_timelogs_on_issue_id” does not exist
ERROR: index “index_term_agreements_on_user_id” does not exist
ERROR: index “index_term_agreements_on_term_id” does not exist
ERROR: index “index_tags_on_name” does not exist
ERROR: index “index_taggings_on_taggable_id_and_taggable_type_and_context” does not exist
ERROR: index “index_taggings_on_taggable_id_and_taggable_type” does not exist
ERROR: index “index_taggings_on_tag_id” does not exist
ERROR: index “index_system_note_metadata_on_note_id” does not exist
ERROR: index “index_subscriptions_on_subscribable_and_user_id_and_project_id” does not exist
ERROR: index “index_snippets_on_visibility_level” does not exist
ERROR: index “index_snippets_on_updated_at” does not exist
ERROR: index “index_snippets_on_title_trigram” does not exist
ERROR: index “index_snippets_on_project_id” does not exist
ERROR: index “index_snippets_on_file_name_trigram” does not exist
ERROR: index “index_snippets_on_author_id” does not exist
ERROR: index “index_services_on_template” does not exist
ERROR: index “index_services_on_project_id” does not exist
ERROR: index “index_sent_notifications_on_reply_key” does not exist
ERROR: index “index_routes_on_source_type_and_source_id” does not exist
ERROR: index “index_routes_on_path_text_pattern_ops” does not exist
ERROR: index “index_routes_on_path” does not exist
ERROR: index “index_resource_label_events_on_user_id” does not exist
ERROR: index “index_resource_label_events_on_merge_request_id” does not exist
ERROR: index “index_resource_label_events_on_label_id” does not exist
ERROR: index “index_resource_label_events_on_issue_id” does not exist
ERROR: index “index_repository_languages_on_project_and_languages_id” does not exist
ERROR: index “index_remote_mirrors_on_project_id” does not exist

<<<<<<<<<<<<<<<<<<<<< FROZEN !!! >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

Here are the processes, which are running on the system at this time:

UID PID PPID C STIME TTY TIME CMD
root 1 0 0 21:48 pts/0 00:00:00 /bin/bash /assets/wrapper
root 15 1 0 21:48 pts/0 00:00:01 runsvdir -P /opt/gitlab/service log: arning: unable to lock directory: /var/log/gitlab/unicorn: temporary failure svlogd: warning: unable to lock directory: /var/log/gitlab/redis: temporary
root 28 15 0 21:48 ? 00:00:00 runsv sshd
root 436 0 0 21:48 pts/1 00:00:00 /bin/sh -c TERM=xterm-256color; export TERM; [ -x /bin/bash ] && ([ -x /usr/bin/script ] && /usr/bin/script -q -c “/bin/bash” /dev/null || exec /bin/bash) || exec /bin/sh
root 442 436 0 21:48 pts/1 00:00:00 /bin/sh -c TERM=xterm-256color; export TERM; [ -x /bin/bash ] && ([ -x /usr/bin/script ] && /usr/bin/script -q -c “/bin/bash” /dev/null || exec /bin/bash) || exec /bin/sh
root 443 442 0 21:48 pts/1 00:00:00 /usr/bin/script -q -c /bin/bash /dev/null
root 444 443 0 21:48 pts/2 00:00:00 sh -c /bin/bash
root 445 444 0 21:48 pts/2 00:00:00 /bin/bash
root 496 15 0 21:49 ? 00:00:00 runsv redis
root 534 15 0 21:49 ? 00:00:00 runsv postgresql
root 673 15 0 21:49 ? 00:00:00 runsv unicorn
root 713 15 0 21:49 ? 00:00:00 runsv sidekiq
git 717 713 1 21:49 ? 00:00:18 sidekiq 5.1.3 gitlab-rails [0 of 25 busy]
root 731 15 0 21:49 ? 00:00:00 runsv gitlab-workhorse
root 752 15 0 21:49 ? 00:00:00 runsv nginx
root 778 15 0 21:49 ? 00:00:00 runsv logrotate
root 793 15 0 21:49 ? 00:00:00 runsv gitaly
root 827 15 0 21:49 ? 00:00:00 runsv node-exporter
root 918 15 0 21:49 ? 00:00:00 runsv gitlab-monitor
root 943 15 0 21:49 ? 00:00:00 runsv redis-exporter
root 963 15 0 21:49 ? 00:00:00 runsv prometheus
root 1000 15 0 21:49 ? 00:00:00 runsv alertmanager
root 1030 15 0 21:49 ? 00:00:00 runsv postgres-exporter
root 1416 963 0 21:49 ? 00:00:00 svlogd -tt /var/log/gitlab/prometheus
root 1545 1 0 21:49 pts/0 00:00:00 /bin/bash /opt/gitlab/bin/gitlab-ctl tail
root 1546 1545 0 21:49 pts/0 00:00:00 /opt/gitlab/embedded/bin/ruby /opt/gitlab/embedded/bin/omnibus-ctl gitlab /opt/gitlab/embedded/service/omnibus-ctl* tail
root 1568 1546 0 21:49 pts/0 00:00:00 sh -c find -L /var/log/gitlab -type f -not -path ‘/sasl/’ | grep -E -v ‘(config|lock|@|gzip|tgz|gz)’ | xargs tail --follow=name --retry
root 1571 1568 0 21:49 pts/0 00:00:00 xargs tail --follow=name --retry
root 1572 1571 0 21:49 pts/0 00:00:00 tail --follow=name --retry /var/log/gitlab/gitlab-shell/gitlab-shell.log /var/log/gitlab/gitlab-rails/production_json.log /var/log/gitlab/gitlab-rails/api_json.log /var/log/gitlab/gitlab-ra
root 2687 0 0 21:50 pts/3 00:00:00 /bin/bash
gitlab-+ 5116 534 0 21:53 ? 00:00:00 /opt/gitlab/embedded/bin/postgres -D /var/opt/gitlab/postgresql/data
gitlab-+ 5128 5116 0 21:53 ? 00:00:00 postgres: checkpointer process
gitlab-+ 5129 5116 0 21:53 ? 00:00:00 postgres: writer process
gitlab-+ 5130 5116 0 21:53 ? 00:00:00 postgres: wal writer process
gitlab-+ 5131 5116 0 21:53 ? 00:00:00 postgres: autovacuum launcher process
gitlab-+ 5132 5116 0 21:53 ? 00:00:00 postgres: stats collector process
git 5612 2687 1 21:53 pts/3 00:00:10 /opt/gitlab/embedded/bin/rake gitlab:backup:restore
gitlab-+ 5728 5116 0 21:54 ? 00:00:00 postgres: gitlab gitlabhq_production [local] idle
git 5902 5612 0 21:54 pts/3 00:00:00 gzip -cd
git 5903 5612 0 21:54 pts/3 00:00:00 psql gitlabhq_production
gitlab-+ 5904 5116 0 21:54 ? 00:00:00 postgres: gitlab gitlabhq_production [local] DROP INDEX
root 14588 0 0 22:04 pts/4 00:00:00 bash
root 14640 14588 0 22:04 pts/4 00:00:00 ps -ef