Okay, so with a fresh outlook and more coffee, I found the problem; I believe there is an error in how gitaly’s config.toml is generated when there isn’t one present. I had assumed that removing the config.toml file and letting gitaly reinstall and introduce it’s own config file would potentially fix issues I had with gitaly but the default config file was generated with a bad bin_dir
root@git:/home/git/gitaly# cat config.toml
#bin_dir = "tmp/tests/gitaly"
bin_dir = "/home/git/gitaly"
..
I commented out the non-existent tmp/tests/gitaly and replaced it with the correct path to gitaly where gitaly-hooks is actually present.
I suspect this problem of finding the bin_dir is introduced in Gitaly’s installation steps, where it checks for a config.toml, and if none exists, creates one with this bad setting. The config.toml.example has the correct information, so I’m not sure where gitaly installation pulled this tmp directory from.
root@git:/home/git/gitaly# cat config.toml.example
# Example Gitaly configuration file
# Documentation lives at https://docs.gitlab.com/ee/administration/gitaly/ and
# https://docs.gitlab.com/ee//administration/gitaly/reference
socket_path = "/home/git/gitlab/tmp/sockets/private/gitaly.socket"
# The directory where Gitaly's executables are stored
bin_dir = "/home/git/gitaly"
I hope this can help someone else who experiences the same problem as me.
As this is likely an installation bug for gitaly that may only apply to source upgrades, how would I raise this as a bug to be tracked?