Restore of repositories fails in gitlab-omni docker image

Community
1

The restore of repositories fails on calling from /opt/gitlab/embedded/service/gitlab-rails/lib/backup/repositories.rb

    override :restore
    def restore(destination_path)
      strategy.start(:restore, destination_path)
      enqueue_consecutive

    ensure
      strategy.finish!

      cleanup_snippets_without_repositories
      restore_object_pools
    end

somewhere in strategy.start(:restore, destination_path)

The error is:

2023-02-16 01:13:51 UTC -- Restoring repositories ...
2023-02-16 01:13:51 +0000 -- Deleting backup and restore lock file
rake aborted!
Errno::EACCES: Permission denied - /var/opt/gitlab/backups
/opt/gitlab/embedded/lib/ruby/2.7.0/open3.rb:218:in `spawn'
/opt/gitlab/embedded/lib/ruby/2.7.0/open3.rb:218:in `popen_run'
/opt/gitlab/embedded/lib/ruby/2.7.0/open3.rb:159:in `popen2'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/gitaly_backup.rb:43:in `start'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/repositories.rb:34:in `restore'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/manager.rb:108:in `run_restore_task'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/manager.rb:261:in `block in run_all_restore_tasks'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/manager.rb:259:in `each_key'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/manager.rb:259:in `run_all_restore_tasks'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/manager.rb:76:in `restore'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:26:in `block (4 levels) in <top (required)>'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:205:in `lock'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:23:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `block in execute'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `execute'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/sentry-ruby-core-5.1.1/lib/sentry/rake.rb:26:in `execute'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:219:in `block in invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `synchronize'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:188:in `invoke'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:160:in `invoke_task'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `block (2 levels) in top_level'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `block in top_level'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:125:in `run_with_threads'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:110:in `top_level'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:83:in `block in run'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:186:in `standard_exception_handling'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:80:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/exe/rake:27:in `<top (required)>'
/opt/gitlab/embedded/bin/rake:23:in `load'
/opt/gitlab/embedded/bin/rake:23:in `<top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/bundler-2.3.15/lib/bundler/cli/exec.rb:58:in `load'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/bundler-2.3.15/lib/bundler/cli/exec.rb:58:in `kernel_load'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/bundler-2.3.15/lib/bundler/cli/exec.rb:23:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/bundler-2.3.15/lib/bundler/cli.rb:483:in `exec'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/bundler-2.3.15/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/bundler-2.3.15/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/bundler-2.3.15/lib/bundler/vendor/thor/lib/thor.rb:392:in `dispatch'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/bundler-2.3.15/lib/bundler/cli.rb:31:in `dispatch'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/bundler-2.3.15/lib/bundler/vendor/thor/lib/thor/base.rb:485:in `start'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/bundler-2.3.15/lib/bundler/cli.rb:25:in `start'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/bundler-2.3.15/exe/bundle:48:in `block in <top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/bundler-2.3.15/lib/bundler/friendly_errors.rb:117:in `with_friendly_errors'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/bundler-2.3.15/exe/bundle:36:in `<top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => gitlab:backup:restore
Transfering ownership of /var/opt/gitlab/gitlab-rails/shared/registry to registry

I’ve tried all kinds of variation of permissions that I could think about, but no luck.
Please help!

2

Sorry, I forgot several details:

  1. The gitlab-omni version is 15.8.1
  2. I’m copying the backup image to /var/opt/gitlab/backups location and running
    gitlab-backup restore BACKUP=${backup} --trace
  3. I also tried to untar the image manually, to disable untar in the code and play with the permissions of the backup files and /var/opt/gitlab/backups.
  4. I ran the restore of everything except repositories and it worked as expected
3

The problem was resolved.
the gitlab_rails[‘backup_gitaly_backup_path’] pointed to the backup directory instead of backup binary. which is gitaly-backup