Backup faild

How to Use GitLab
1

Hi All,

Last few days backup is not worked. I am running backup command in crontab at night. Some time it will work and some time it will not work. I tried to run it manually and same thing is happening. Please find the below details and help me to resolve it.

Version : GitLab Community Edition 11.1.2
backup command : /opt/gitlab/bin/gitlab-rake gitlab:backup:create
total size of backup if completed : 40GB
Error message when it faild :
[SKIPPED] Wiki

  • Program-Management/Program-Management … rake aborted!
    GRPC::Core::CallError: call#run_batch failed somehow
    /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/grpc-1.11.0-x86_64-linux/src/ruby/lib/grpc/generic/active_call.rb:253:in run_batch' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/grpc-1.11.0-x86_64-linux/src/ruby/lib/grpc/generic/active_call.rb:253:in remote_read’
    /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/grpc-1.11.0-x86_64-linux/src/ruby/lib/grpc/generic/active_call.rb:326:in block in each_remote_read_then_finish' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/grpc-1.11.0-x86_64-linux/src/ruby/lib/grpc/generic/active_call.rb:325:in loop’
    /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/grpc-1.11.0-x86_64-linux/src/ruby/lib/grpc/generic/active_call.rb:325:in each_remote_read_then_finish' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client/repository_service.rb:348:in each’
    /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client/repository_service.rb:348:in block in gitaly_fetch_stream_to_file' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client/repository_service.rb:347:in open’
    /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client/repository_service.rb:347:in gitaly_fetch_stream_to_file' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client/repository_service.rb:201:in create_bundle’
    /opt/gitlab/embedded/service/gitlab-rails/lib/backup/repository.rb:67:in backup_project_gitaly' /opt/gitlab/embedded/service/gitlab-rails/lib/backup/repository.rb:53:in block in backup_project’
    /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client.rb:289:in block (2 levels) in migrate' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client.rb:329:in allow_n_plus_1_calls’
    /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client.rb:282:in block in migrate' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/influx_db.rb:98:in measure’
    /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client.rb:280:in migrate' /opt/gitlab/embedded/service/gitlab-rails/lib/backup/repository.rb:285:in gitaly_migrate’
    /opt/gitlab/embedded/service/gitlab-rails/lib/backup/repository.rb:51:in backup_project' /opt/gitlab/embedded/service/gitlab-rails/lib/backup/repository.rb:27:in block in dump’
    /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/relation/batches.rb:51:in block (2 levels) in find_each' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/relation/batches.rb:51:in each’
    /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/relation/batches.rb:51:in block in find_each' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/relation/batches.rb:124:in find_in_batches’
    /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/relation/batches.rb:50:in find_each' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/querying.rb:9:in find_each’
    /opt/gitlab/embedded/service/gitlab-rails/lib/backup/repository.rb:17:in dump' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:80:in block (4 levels) in <top (required)>’
    /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/task.rb:271:in block in execute' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/task.rb:271:in each’
    /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/task.rb:271:in execute' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/task.rb:213:in block in invoke_with_call_chain’
    /opt/gitlab/embedded/lib/ruby/2.4.0/monitor.rb:214:in mon_synchronize' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/task.rb:193:in invoke_with_call_chain’
    /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/task.rb:182:in invoke' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:11:in block (3 levels) in <top (required)>’
    /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/task.rb:271:in block in execute' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/task.rb:271:in each’
    /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/task.rb:271:in execute' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/task.rb:213:in block in invoke_with_call_chain’
    /opt/gitlab/embedded/lib/ruby/2.4.0/monitor.rb:214:in mon_synchronize' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/task.rb:193:in invoke_with_call_chain’
    /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/task.rb:182:in invoke' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/application.rb:160:in invoke_task’
    /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/application.rb:116:in block (2 levels) in top_level' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/application.rb:116:in each’
    /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/application.rb:116:in block in top_level' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/application.rb:125:in run_with_threads’
    /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/application.rb:110:in top_level' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/application.rb:83:in block in run’
    /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/application.rb:186:in standard_exception_handling' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/application.rb:80:in run’
    /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/exe/rake:27:in <top (required)>' /opt/gitlab/embedded/bin/rake:23:in load’
    /opt/gitlab/embedded/bin/rake:23:in <top (required)>' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/cli/exec.rb:74:in load’
    /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/cli/exec.rb:74:in kernel_load' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/cli/exec.rb:28:in run’
    /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/cli.rb:424:in exec' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/vendor/thor/lib/thor/command.rb:27:in run’
    /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/vendor/thor/lib/thor/invocation.rb:126:in invoke_command' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/vendor/thor/lib/thor.rb:387:in dispatch’
    /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/cli.rb:27:in dispatch' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/vendor/thor/lib/thor/base.rb:466:in start’
    /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/cli.rb:18:in start' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/exe/bundle:30:in block in <top (required)>’
    /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/friendly_errors.rb:124:in with_friendly_errors' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/exe/bundle:22:in <top (required)>’
    /opt/gitlab/embedded/bin/bundle:23:in load' /opt/gitlab/embedded/bin/bundle:23:in ’
    Tasks: TOP => gitlab:backup:repo:create
    root@vagit10:/var/opt/gitlab/backups#