Hi there,
I’m running Gitlab EE 11.1.2-ee.0 from the Debian package, on a Debian 9 host. I’m getting periodic problems with backups. The output here is from gitlab-rake gitlab:backup:create --trace
Firstly I’m getting timeout warnings from some of our bigger repositories (forks of the Linux kernel).
[WARNING] Executing 4:Deadline Exceeded
Ignoring error on project/kernel - Failed to backup repo
Then while backing up another kernel repo, I get the following:
* otherproject/kernel ... rake aborted! GRPC::Core::CallError: call#run_batch failed somehow /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/grpc-1.11.0-x86_64-linux/src/ruby/lib/grpc/generic/active_call.rb:253:in `run_batch' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/grpc-1.11.0-x86_64-linux/src/ruby/lib/grpc/generic/active_call.rb:253:in `remote_read' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/grpc-1.11.0-x86_64-linux/src/ruby/lib/grpc/generic/active_call.rb:326:in `block in each_remote_read_then_finish' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/grpc-1.11.0-x86_64-linux/src/ruby/lib/grpc/generic/active_call.rb:325:in `loop' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/grpc-1.11.0-x86_64-linux/src/ruby/lib/grpc/generic/active_call.rb:325:in `each_remote_read_then_finish' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client/repository_service.rb:348:in `each' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client/repository_service.rb:348:in `block in gitaly_fetch_stream_to_file' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client/repository_service.rb:347:in `open' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client/repository_service.rb:347:in `gitaly_fetch_stream_to_file' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client/repository_service.rb:201:in `create_bundle' /opt/gitlab/embedded/service/gitlab-rails/lib/backup/repository.rb:67:in `backup_project_gitaly' /opt/gitlab/embedded/service/gitlab-rails/lib/backup/repository.rb:53:in `block in backup_project' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client.rb:289:in `block (2 levels) in migrate' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client.rb:329:in `allow_n_plus_1_calls' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client.rb:282:in `block in migrate' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/influx_db.rb:98:in `measure' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client.rb:280:in `migrate' /opt/gitlab/embedded/service/gitlab-rails/lib/backup/repository.rb:285:in `gitaly_migrate' /opt/gitlab/embedded/service/gitlab-rails/lib/backup/repository.rb:51:in `backup_project' /opt/gitlab/embedded/service/gitlab-rails/lib/backup/repository.rb:27:in `block in dump' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/relation/batches.rb:51:in `block (2 levels) in find_each' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/relation/batches.rb:51:in `each' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/relation/batches.rb:51:in `block in find_each' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/relation/batches.rb:124:in `find_in_batches' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/relation/batches.rb:50:in `find_each' /opt/gitlab/embedded/service/gitlab-rails/lib/backup/repository.rb:17:in `dump' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:80:in `block (4 levels) in ' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/task.rb:271:in `block in execute' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/task.rb:271:in `each' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/task.rb:271:in `execute' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/task.rb:213:in `block in invoke_with_call_chain' /opt/gitlab/embedded/lib/ruby/2.4.0/monitor.rb:214:in `mon_synchronize' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/task.rb:193:in `invoke_with_call_chain' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/task.rb:182:in `invoke' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:11:in `block (3 levels) in ' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/task.rb:271:in `block in execute' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/task.rb:271:in `each' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/task.rb:271:in `execute' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/task.rb:213:in `block in invoke_with_call_chain' /opt/gitlab/embedded/lib/ruby/2.4.0/monitor.rb:214:in `mon_synchronize' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/task.rb:193:in `invoke_with_call_chain' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/task.rb:182:in `invoke' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/application.rb:160:in `invoke_task' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/application.rb:116:in `block (2 levels) in top_level' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/application.rb:116:in `each' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/application.rb:116:in `block in top_level' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/application.rb:125:in `run_with_threads' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/application.rb:110:in `top_level' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/application.rb:83:in `block in run' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/application.rb:186:in `standard_exception_handling' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/application.rb:80:in `run' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/exe/rake:27:in `' /opt/gitlab/embedded/bin/rake:23:in `load' /opt/gitlab/embedded/bin/rake:23:in `' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/cli/exec.rb:74:in `load' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/cli/exec.rb:74:in `kernel_load' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/cli/exec.rb:28:in `run' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/cli.rb:424:in `exec' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/vendor/thor/lib/thor/invocation.rb:126:in `invoke_command' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/vendor/thor/lib/thor.rb:387:in `dispatch' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/cli.rb:27:in `dispatch' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/vendor/thor/lib/thor/base.rb:466:in `start' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/cli.rb:18:in `start' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/exe/bundle:30:in `block in ' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/friendly_errors.rb:124:in `with_friendly_errors' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/exe/bundle:22:in `' /opt/gitlab/embedded/bin/bundle:23:in `load' /opt/gitlab/embedded/bin/bundle:23:in `' Tasks: TOP => gitlab:backup:repo:create
Any idea what’s causing this or how I can get backups to complete reliably?