Summary
We have the following error during a “Unpacking backup” step of the restore of backup:
Starting restore 2023-02-20 11:04:33 +0000 – Unpacking backup … rake aborted! ActiveRecord::QueryCanceled: PG::QueryCanceled: ERROR: canceling statement due to statement timeout
Steps to reproduce
Run the command: “gitlab-backup restore BACKUP= force=yes” With the docker: gitlab/gitlab-ce:15.4.0-ce.0
What is the current bug behavior?
The backup restore is aborted and restore is failed.
What is the expected correct behavior?
The backup restore is successful and Gitlab is restored.
Relevant logs and/or screenshots
Starting restore
2023-02-20 11:04:33 +0000 – Unpacking backup …
rake aborted!
ActiveRecord::QueryCanceled: PG::QueryCanceled: ERROR: canceling statement due to statement timeout
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:120:in block in write_using_load_balancer' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:115:in block in read_write’
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:184:in retry_with_backoff' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:111:in read_write’
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:119:in write_using_load_balancer' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:89:in method_missing’
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:76:in block (2 levels) in drop_tables' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:76:in each’
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:76:in block in drop_tables' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/each_database.rb:17:in block (2 levels) in each_database_connection’
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/each_database.rb:77:in block in with_shared_connection' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/shared_model.rb:34:in using_connection’
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/each_database.rb:74:in with_shared_connection' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/each_database.rb:16:in block in each_database_connection’
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/each_database.rb:11:in each_pair' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/each_database.rb:11:in each_database_connection’
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:59:in drop_tables' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:46:in block (3 levels) in <top (required)>’
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/database.rb:86:in restore' /opt/gitlab/embedded/service/gitlab-rails/lib/backup/manager.rb:101:in run_restore_task’
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/manager.rb:244:in block in run_all_restore_tasks' /opt/gitlab/embedded/service/gitlab-rails/lib/backup/manager.rb:242:in each’
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/manager.rb:242:in run_all_restore_tasks' /opt/gitlab/embedded/service/gitlab-rails/lib/backup/manager.rb:76:in restore’
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:20:in block (3 levels) in <top (required)>' /opt/gitlab/embedded/bin/bundle:23:in load’
/opt/gitlab/embedded/bin/bundle:23:in `’
Caused by:
PG::QueryCanceled: ERROR: canceling statement due to statement timeout
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:120:in block in write_using_load_balancer' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:115:in block in read_write’
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:184:in retry_with_backoff' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:111:in read_write’
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:119:in write_using_load_balancer' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:89:in method_missing’
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:76:in block (2 levels) in drop_tables' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:76:in each’
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:76:in block in drop_tables' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/each_database.rb:17:in block (2 levels) in each_database_connection’
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/each_database.rb:77:in block in with_shared_connection' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/shared_model.rb:34:in using_connection’
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/each_database.rb:74:in with_shared_connection' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/each_database.rb:16:in block in each_database_connection’
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/each_database.rb:11:in each_pair' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/each_database.rb:11:in each_database_connection’
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:59:in drop_tables' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:46:in block (3 levels) in <top (required)>’
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/database.rb:86:in restore' /opt/gitlab/embedded/service/gitlab-rails/lib/backup/manager.rb:101:in run_restore_task’
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/manager.rb:244:in block in run_all_restore_tasks' /opt/gitlab/embedded/service/gitlab-rails/lib/backup/manager.rb:242:in each’
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/manager.rb:242:in run_all_restore_tasks' /opt/gitlab/embedded/service/gitlab-rails/lib/backup/manager.rb:76:in restore’
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:20:in block (3 levels) in <top (required)>' /opt/gitlab/embedded/bin/bundle:23:in load’
/opt/gitlab/embedded/bin/bundle:23:in `’
Tasks: TOP => gitlab:db:drop_tables
(See full trace by running task with --trace)
2023-02-20 11:20:43 +0000 – Unpacking backup … done
2023-02-20 11:20:43 +0000 – Restoring main_database …
2023-02-20 11:20:43 UTC – Cleaning the database …
2023-02-20 11:22:06 +0000 – Deleting tar staging files …
2023-02-20 11:22:06 +0000 – Cleaning up /var/opt/gitlab/backups/backup_information.yml
2023-02-20 11:22:06 +0000 – Cleaning up /var/opt/gitlab/backups/db
2023-02-20 11:22:06 +0000 – Cleaning up /var/opt/gitlab/backups/repositories
2023-02-20 11:22:11 +0000 – Cleaning up /var/opt/gitlab/backups/uploads.tar.gz
2023-02-20 11:22:11 +0000 – Cleaning up /var/opt/gitlab/backups/builds.tar.gz
2023-02-20 11:22:11 +0000 – Cleaning up /var/opt/gitlab/backups/artifacts.tar.gz
2023-02-20 11:22:11 +0000 – Cleaning up /var/opt/gitlab/backups/pages.tar.gz
2023-02-20 11:22:11 +0000 – Cleaning up /var/opt/gitlab/backups/lfs.tar.gz
2023-02-20 11:22:11 +0000 – Cleaning up /var/opt/gitlab/backups/terraform_state.tar.gz
2023-02-20 11:22:11 +0000 – Cleaning up /var/opt/gitlab/backups/packages.tar.gz
2023-02-20 11:22:11 +0000 – Deleting tar staging files … done
2023-02-20 11:22:11 +0000 – Deleting backups/tmp …
2023-02-20 11:22:11 +0000 – Deleting backups/tmp … done