Fail: 14: failed to connect to all addresses and 404 page not found

Hi all,
Is anyone able to help with an issue that we have been having all of a sudden. we had an issue where the server ran out of disk space. After some space was freed up and GitLab was restarted, we can no longer view any repositories. As an admin I can see all the repos and the data is there, but when I click on the project I just get a page 404 error, page not found and I can’t seem to resolve it.

Running GitLab Omni 13.2.4.

Running a gitlab-rake gitlab:check I get an error Gitaly FAIL: 14: failed to connect to all addresses.

Running a reconfigure and a restart doesn’t seem to fix anything.

has anyone got any ideas on where to start looking at this. The server seems to be running a part from the these pages. If log files would help then let me know which particular ones might be able to provide pointers.

Hopefully someone in the hive mind may have a few pointers.

Cheers,
Alan

I’d start by checking Gitaly service logs under /var/log/gitlab/gitaly/, branching off of the specific gitlab:check error you received about not being able to connect onto a/the Gitaly service. Gitaly is also the service used to serve repository content such as trees and files, so it is a good starting point given your issue.

You may also find it easier to tail logs as you reproduce issues, so it gets a little easier in spotting errors and potential stack-traces that you can then lookup (or share here): https://docs.gitlab.com/omnibus/settings/logs.html#tail-logs-in-a-console-on-the-server

Star, Thanks for this. I was looking at the log and kept seeing this line.

time=“2020-12-01T12:58:37Z” level=fatal msg=“load config” config_path=/var/opt/gitlab/gitaly/config.toml error=“stat /opt/gitlab/embedded/service/gitaly-ruby/git-hooks/pre-receive: no such file or directory”

I did have a look and saw the file had been renamed to Xpre-receive. I didn’t think much of it as the change was made months ago, but knowing the users it could well have been changed and GitLab not restarted.

Hopefully, that’s sorted it till someone changes it again.