Backup restore fails restoring custom hooks

Hi,

I’m currently trying to restore a backup to a testing environment. Everything seems to go well until custom hooks get restored, at which point it throws the following error:

@hashed/13/8d/138d9e809e386a7b800791d1f664f56d1c55f3d1ba411b950862729bc486c5ce.git (group/git-repo): manager: restore custom hooks, "@hashed/13/8d/138d9e809e386a7b800791d1f664f56d1c55f3d1ba411b950862729bc486c5ce/1684932034_2023_05_24_15.11.3/001.custom_hooks.tar": rpc error: code = Internal desc = setting custom hooks: generating prepared vote: walking directory: opening file: open /opt/git-data/repositories/+gitaly/tmp/default-repositories.old.1685449481.2544498313/custom_hooks/post-receive: no such file or directory\n -

It throws above error for a few other repositories, strangely they’re all in the same group, but only one has custom hooks set. I can gitlab-ctl reconfigure the machine afterwards, at which point everything seems to work except the repositories with custom hooks set. What I did was follow the backup/restore guide.

Version: 15.11.3 / Omnibus

Does anyone have any idea where I could be looking to solve this?

I’ve just hit this myself. I’m guessing it’s related to this from this.

Our hooks are managed by a sidecar container, so they’re actually symlinks to a central ‘gitconfigs’ project that’s normally available to gitlab as a container volume. However during restore that destination path doesn’t exist yet because we haven’t run the sidecar yet (because it needs the gitconfigs repository to check out)

If you look for custom_hooks in the backup tarball you’ll see a 001.custom_hooks.tar for each repository that had hooks configured. When I look at one of those I see it contains the dangling symlink:

tar -tvf repositories/@hashed/1a/d2/1ad269a743bd01b5bb74f135c332a4acc98ef1a570d966fcd6a801de6d9ae3bc/1685931329_2023_06_05_15.11.3-ee/001.custom_hooks.tar
drwxr-sr-x git/git           0 2021-08-26 03:45 custom_hooks/
drwxr-sr-x git/git           0 2021-08-26 03:45 custom_hooks/post-receive.d/
lrwxrwxrwx git/git           0 2021-08-26 03:45 custom_hooks/post-receive.d/0-post-receive-push-to-codecommit -> /var/opt/custom-gitlab-metadata/gitconfigs/hooks/post-receive-push-to-codecommit

… because the sidecar itself creates and manages /var/opt/custom-gitlab-metadata so without that running at the point of restore the target will be missing, so the walking directory: opening file: open logic fails.