Cannot create GitLab backup

Hi,

My self-managed GitLab version is GitLab CE 14.8.2.

I’ve tried the “gitlab-rake gitlab:backup:create GZIP_RSYNCABLE=yes --trace” command to create the GitLab backup.

But it returns the following messages and the backup is not created.

Backup Begin Time: 2022-03-11 17:37:17
ok: down: gitaly: 0s, normally up
2022-03-11 17:37:32 +0800 -- Dumping database ... 
Dumping PostgreSQL database gitlabhq_production ... [DONE]
2022-03-11 17:37:35 +0800 -- done
2022-03-11 17:37:35 +0800 -- Dumping repositories ... 
{"command":"create","gl_project_path":"gitlab-instance-1de5cc52/monitoring","level":"info","msg":"started create","relative_path":"@hashed/6b/86/6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b.git","storage_name":"default","time":"2022-03-11T09:37:35.964Z"}
{"command":"create","error":"manager: isEmpty: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial unix /var/opt/gitlab/gitaly/gitaly.socket: connect: no such file or directory\"","gl_project_path":"gitlab-instance-1de5cc52/monitoring","level":"error","msg":"create failed","relative_path":"@hashed/6b/86/6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b.git","storage_name":"default","time":"2022-03-11T09:37:35.965Z"}
{"command":"create","gl_project_path":"gitlab-instance-1de5cc52/monitoring.wiki","level":"info","msg":"started create","relative_path":"@hashed/6b/86/6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b.wiki.git","storage_name":"default","time":"2022-03-11T09:37:35.991Z"}

               ... skip messages ...

{"command":"create","error":"manager: isEmpty: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial unix /var/opt/gitlab/gitaly/gitaly.socket: connect: no such file or directory\"","gl_project_path":"ta/ta/ta_toolbox.wiki","level":"error","msg":"create failed","relative_path":"@hashed/0b/91/0b918943df0962bc7a1824c0555a389347b4febdc7cf9d1254406d80ce44e3f9.wiki.git","storage_name":"default","time":"2022-03-11T09:37:36.019Z"}
{"command":"create","gl_project_path":"ta/ta/ta_toolbox","level":"info","msg":"started create","relative_path":"@hashed/0b/91/0b918943df0962bc7a1824c0555a389347b4febdc7cf9d1254406d80ce44e3f9.design.git","storage_name":"default","time":"2022-03-11T09:37:36.020Z"}
{"command":"create","error":"manager: isEmpty: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial unix /var/opt/gitlab/gitaly/gitaly.socket: connect: no such file or directory\"","gl_project_path":"ta/ta/ta_toolbox","level":"error","msg":"create failed","relative_path":"@hashed/0b/91/0b918943df0962bc7a1824c0555a389347b4febdc7cf9d1254406d80ce44e3f9.design.git","storage_name":"default","time":"2022-03-11T09:37:36.020Z"}
{
	"level": "fatal",
	"msg": "create: pipeline: 27 failures encountered:\n - @hashed/6b/86/6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b.git (gitlab-instance-1de5cc52/monitoring): manager: isEmpty: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial unix /var/opt/gitlab/gitaly/gitaly.socket: connect: no such file or directory\"\n - @hashed/6b/86/6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b.wiki.git (gitlab-instance-1de5cc52/monitoring.wiki): manager: isEmpty: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial unix /var/opt/gitlab/gitaly/gitaly.socket: connect: no such file or directory\"\n - @hashed/6b/86/6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b.design.git (gitlab-instance-1de5cc52/monitoring): manager: isEmpty: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial unix /var/opt/gitlab/gitaly/gitaly.socket: connect: no such file or directory\"\n - @hashed/45/23/4523540f1504cd17100c4835e85b7eefd49911580f8efff0599a8f283be6b9e3.git (ta/tclkernel_BAK): manager: isEmpty: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial unix /var/opt/gitlab/gitaly/gitaly.socket: connect: no such file or directory\"\n - @hashed/45/23/4523540f1504cd17100c4835e85b7eefd49911580f8efff0599a8f283be6b9e3.wiki.git (ta/tclkernel_BAK.wiki): manager: isEmpty: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial unix /var/opt/gitlab/gitaly/gitaly.socket: connect: no such file or directory\"\n - @hashed/45/23/4523540f1504cd17100c4835e85b7eefd49911580f8efff0599a8f283be6b9e3.design.git (ta/tclkernel_BAK): manager: isEmpty: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial unix /var/opt/gitlab/gitaly/gitaly.socket: connect: no such file or directory\"\n - @hashed/35/13/35135aaa6cc23891b40cb3f378c53a17a1127210ce60e125ccf03efcfdaec458.git (ta/tabin): manager: isEmpty: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial unix /var/opt/gitlab/gitaly/gitaly.socket: connect: no such file or directory\"\n - @hashed/35/13/35135aaa6cc23891b40cb3f378c53a17a1127210ce60e125ccf03efcfdaec458.wiki.git (ta/tabin.wiki): manager: isEmpty: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial unix /var/opt/gitlab/gitaly/gitaly.socket: connect: no such file or directory\"\n - @hashed/35/13/35135aaa6cc23891b40cb3f378c53a17a1127210ce60e125ccf03efcfdaec458.design.git (ta/tabin): manager: isEmpty: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial unix /var/opt/gitlab/gitaly/gitaly.socket: connect: no such file or directory\"\n - @hashed/62/4b/624b60c58c9d8bfb6ff1886c2fd605d2adeb6ea4da576068201b6c6958ce93f4.git (ta/ta_toolbox): manager: isEmpty: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial unix /var/opt/gitlab/gitaly/gitaly.socket: connect: no such file or directory\"\n - @hashed/62/4b/624b60c58c9d8bfb6ff1886c2fd605d2adeb6ea4da576068201b6c6958ce93f4.wiki.git (ta/ta_toolbox.wiki): manager: isEmpty: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial unix /var/opt/gitlab/gitaly/gitaly.socket: connect: no such file or directory\"\n - @hashed/62/4b/624b60c58c9d8bfb6ff1886c2fd605d2adeb6ea4da576068201b6c6958ce93f4.design.git (ta/ta_toolbox): manager: isEmpty: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial unix /var/opt/gitlab/gitaly/gitaly.socket: connect: no such file or directory\"\n",
	"pid": 22265,
	"time": "2022-03-11T09:37:36.038Z"
}

Before upgrading to v14.8.2, I cloned the host and the GitLab version on the cloned host is 14.1.8.

Surprisingly, the backup can be created on the cloned host with version 14.1.8.
And I found this issue also happens on CE v14.6.1

Does anyone know how to resolve this backup problem? What is the root cause while creating v14.8.2 backup?

Was your output cut short? I don’t remember what output that rake task is supposed to create, but I believe it’s more than you show. Furthermore there are no serious errors in the output you show (all the errors reported are empty repositories).

Hi @grove , yes, the output I show is cut short, the whole output is very long for all repositories.

How can I backup successfully in CE 14.8.2 since there are empty repositories and backup can be created in CE 14.1.8?

It’s a bad idea to cut the output, you might cut away something important. If you do want to cut, show the last part, that is often where errors are.

Our GitLab is still not on 14.8, but up to 14.7 your command for taking a backup works, and I haven’t heard anything suggesting it should be the same on 14.8. I’m guessing we have some empty repositories among the ~4300 we have.

Hi @grove , I’ve updated the error messages so that you can know the head and the tail of the error messages. Thanks.

Hi all. Any thoughts? Cheers

The same problem with gitlab 15.0.5.
If I stopped gitaly, which is recommended in gitlab documentation on backup

, I got this error about “no such file for gitaly.socket”.

Also It may be related to

I have a feeling that I’ve seen somewhere that you in some cases (possibly when gitaly is running somewhere else) can’t take a backup when gitaly is stopped.