I tried to create a backup, but I get an error on one of my repositories
gitlab-rake gitlab:backup:create
* xxxx ... [WARNING] Executing 13:CreateBundle: cmd wait failed: exit status 1
Ignoring error on xxx - Failed to backup repo
I’ve checked on stackoveflow and in this forum, but can’t find a precise match.
There is this issue, which is fixed in 11.1.4 https://gitlab.com/gitlab-org/gitlab-ce/issues/49538,
could this issue be related, as it was about timeouts, although this looks to me
like CreateBundle actually failed and wait(1) picked up a status of 1 for the command.
Thanks in advance for any tips/pointers/solutions
Here are the version details
root@gitlab:~# gitlab-rake gitlab:env:info
System information
System: Ubuntu 16.04
Current User: git
Using RVM: no
Ruby Version: 2.4.4p296
ssh-rsa
Bundler Version:1.16.2
Rake Version: 12.3.1
Redis Version: 3.2.11
Git Version: 2.17.1
Sidekiq Version:5.1.3
Go Version: unknown
GitLab information
Version: 11.1.2
Revision: 35936b0
Directory: /opt/gitlab/embedded/service/gitlab-rails
DB Adapter: postgresql
URL: https://xxxx
HTTP Clone URL: https://xxxx/some-group/some-project.git
SSH Clone URL: git@xxxx:some-group/some-project.git
Using LDAP: no
Using Omniauth: yes
Omniauth Providers: saml
GitLab Shell
Version: 7.1.4
Repository storage paths:
- default: /var/opt/gitlab/git-data/repositories
Hooks: /opt/gitlab/embedded/service/gitlab-shell/hooks
Git: /opt/gitlab/embedded/bin/git
I can confirm that this is still happening as of 11.3.0-ee so it was apparently not fixed in 11.1.4. The error message when trying to create a backup is identical to the original poster. Below is my system info.
# gitlab-rake gitlab:env:info
System information
System: Ubuntu 16.04
Proxy: no
Current User: git
Using RVM: no
Ruby Version: 2.4.4p296
Gem Version: 2.7.6
Bundler Version:1.16.2
Rake Version: 12.3.1
Redis Version: 3.2.11
Git Version: 2.18.0
Sidekiq Version:5.1.3
Go Version: unknown
GitLab information
Version: 11.3.0-ee
Revision: 4e822ac
Directory: /opt/gitlab/embedded/service/gitlab-rails
DB Adapter: postgresql
DB Version: 9.6.8
URL: https://xxxx
HTTP Clone URL: https://xxxx/some-group/some-project.git
SSH Clone URL: git@xxxx:some-group/some-project.git
Elasticsearch: no
Geo: no
Using LDAP: no
Using Omniauth: no
GitLab Shell
Version: 8.3.3
Repository storage paths:
- default: /var/opt/gitlab/git-data/repositories
Hooks: /opt/gitlab/embedded/service/gitlab-shell/hooks
Git: /opt/gitlab/embedded/bin/git
Since I noticed that the system info was checking for golang, I decided to install that and retry. This did not make any difference. The error message was the same when attempting to create a backup. For posterity, here is the only line in my system info which changed after installing golang.
Go Version: go1.6.2 linux/amd64
I can see that the backup command is running /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake. There appears to be a call to dump of the Repository class, which I presume is the one found in /opt/gitlab/embedded/service/gitaly-ruby/lib/gitlab/git/repository.rb however this is where the trail goes cold, as there’s no dump method defined here. My best guess is that the method is inherited from some parent class.
I also tried grepping /opt/gitlab/embedded/service/ for things like “exit status” to find where the error is coming from, but did not have any luck there either. If anyone knows where this error message is coming from, I’d be happy to look at the code and try to help get this resolved.
Since this is top search engine result for this error message, I’ll explain how I worked around this issue.
First, it is important to note that only one of the repos was failing to be backed up, the rest did not have any errors. It’s also worth mentioning that this was the largest repo at around 400MB. Finally, the GitLab server was a VM and the host is suspected to have faulty memory (still looking into that).
I debugged some more of the code and it appeared to be a problem when it was looping over the create_bundle result and writing message.data to a file. My best guess is that there was some integrity check somewhere which failed due to memory errors (hardware issue) and that’s what was causing the failure. This is purely speculative, as the error message did not indicate what the problem was, nor did I find any code which was doing integrity checking. It’s just that odds of a memory error happening would have been larger on the 400MB repo than the 128MB repo.
Lets move on to how I worked around the issue, and I warn you up front, it’s not pretty. I updated to the latest version (11.3.1 at the time or writing) and tried again. It failed in the same way. Next, I created a new VM, installed a fresh copy of GitLab using the omnibus installer. I then restored from the backup which had errors. It restored everything except that one repo. Everything else worked fine: the users, passwords, SSH keys, other repos, and so on. I then created a new repo using the web UI. Once that was done, I pushed the copy of the repo I checked out to the server and that worked fine. I then attempted to make a backup and it worked fine.
Differences between the old instance, where the backup failed and the new instance where the backup succeeded:
Different physical machines
New instance was Ubuntu 18.04
3GB RAM instead of 4GB
3 CPUs instead of 4 CPUs
However, it was the same version of GitLab, same versions of the repos, and so on. Hopefully this will help someone in the future who runs into this error. If a developer is ever able to track down the actual issue here, I think we would all appreciate some additional info in the error message as to what is failing so we can more effectively troubleshoot in the future. Given that I don’t have instructions on how to reproduce the error, I think this one is probably going to remain a mystery, but at least now people have a documented workaround (create a new instance and import the failing VM).
Just wanted to add to this a little bit, since I was having the same problem: when running a gitlab-rake gitlab:backup:create, I was getting the CreateBundle error. I happened to look at /var/log/messages (I’m running RHEL6) and found that I was getting an “Out of memory” condition, and the git process was being killed by the OS. Increasing the amount of swap space available to the system let the process continue to completion and the backup finished OK. The repository that was failing was our largest repo by far, so that was certainly the issue.
I had the same issue. Only one repo failed to create a backup. The whole backup could be restored without that repo. Having read this thread (especially the suggested solution by todpike) I tried to free some memory by killing services as I don’t have swap on that machine. That solved the problem.
There should be an adequate error message about low memory. Probably there are not so many low resource settings of gitlab so that this does not come up often. I was migrating to a faster and bigger server to get gitlab running adequately after having used it for some years on the smaller machine. But it’s not very nice to get stuck on the progress of migration especially when exporting a single project is tedious.
The original (virtual) machine has 5.8GB of memory, no swap, 2 cores.
The failing repository has 4 commits and about 800MB size.
Gitlab version 13.5.1.
I also had the same problem.
The backup was only failing in the largest repository at around 1.9G! (the other from 1.7G had no problems)
The problem was solved by increasing the VM memory from 4G to 6G (I use KVM)
the my System information
System: Ubuntu 20.04
Proxy: no
Current User: git
Using RVM: no
Ruby Version: 2.4.4p296
Gem Version: 2.7.6
Bundler Version:1.16.2
Rake Version: 12.3.1
Redis Version: 3.2.11
Git Version: 2.18.0
Sidekiq Version:5.1.3
Go Version: unknown
GitLab information
Version: 11.2.3-ee
Revision: aadca994104
Directory: /opt/gitlab/embedded/service/gitlab-rails
DB Adapter: postgresql
DB Version: 9.6.8
URL: https://xxxx
HTTP Clone URL: https://xxxxx/some-group/some-project.git
SSH Clone URL: git@xxxx:some-group/some-project.git
Elasticsearch: no
Geo: no
Using LDAP: yes
Using Omniauth: no