After upgrading gitlab about two weeks ago, I noticed the other day that the scheduled backup task had stopped working. After running it manually, I see the following errors (edited slightly for readability):
time="2021-10-20T14:59:07Z" level=error msg="create failed" command=create
error="manager: write bundle: *backup.FilesystemSink write: write file
\"/srv/gitlab/tmp/backups/repositories/@hashed/58/eb/58eb0dd988df36c54
ae88fcf58afcd0a69944c335d82dfbe68094063f54f4363.bundle\": rpc error:
code = Internal desc = cmd wait failed: exit status 1, stderr: \"fatal: bad
object 4b463ea837e76aec593dd95eeb63412347910b3b\\n
error: pack-objects died\\n\"" gl_project_path=(some repo)
relative_path=@hashed/58/eb/58eb0dd988df36c54ae88fcf58afcd0a
69944c335d82dfbe68094063f54f4363.git storage_name=default
(...)
time="2021-10-20T14:59:50Z" level=fatal msg="create: pipeline: 3 failures encountered"
rake aborted!
Backup::Error: gitaly-backup exit status 1
/srv/gitlab/lib/backup/gitaly_backup.rb:44:in `wait'
/srv/gitlab/lib/backup/repositories.rb:43:in `dump'
/srv/gitlab/lib/tasks/gitlab/backup.rake:114:in `block (4 levels) in <top (required)>'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/rake-13.0.6/exe/rake:27:in `<top (required)>'
/srv/gitlab/bin/bundle:5:in `load'
/srv/gitlab/bin/bundle:5:in `<main>'
Tasks: TOP => gitlab:backup:repo:create
(See full trace by running task with --trace)
The conventional wisdom around pack-objects died that I’ve found mostly relates to resource allocations, namely to increase the amount of memory requested by helpers in the helm chart. I’ll try increasing it, but it’s already at 8gb, which I thought would be plenty.
If anyone has any information or suggestions, please let me know. Thanks!