503 error and Gitaly 14:failed to connect to all addresses

I’m seeing a 503 whenever I try to access any project after the latest upgrade.

Strangely enough I have another instance of Gitlab and after the upgrade that works without issues …

Bug Report: https://gitlab.com/gitlab-org/gitlab/-/issues/232895

Does anybody have this kind of issues? I think I saw somewhere some users fixing this kind of error by changing their DNS servers in /etc/resolv.conf or possibly also /etc/hosts. To me that didn’t help unfortunately :frowning:.

Hi, I had the same problem, but in my case it was related to file permissions: check permissions on /var/opt/gitlab/{git-data|gitlab-ci|gitlab-rails}/*

Thank you for your reply. Which permissions should I have exactly? I think it showed owner=git group=root when I checked. Now everything is in the water since the NAS experienced a multiple drive failure so I cannot check for now …

I my case I moved the data to a NFS mount using this guide

https://docs.gitlab.com/ee/administration/nfs.html

but when I copied the files I forgot to keep the ownership and permission information, but I had a copy so I checked and in my case it where:

git-data:
drwx------ 3 git root 3 ago 4 10:59 .
drwxr-xr-x 24 root root 4096 ago 4 14:19 …
drwxrws— 510 git git 511 ago 4 15:10 repositories

git-data/repositories:

drwxr-s— 2 git git 2 ago 4 10:59 [dirname]
drwxr-s— 2 git git 2 ago 4 10:59 [dirname]
drwxr-s— 6 git git 6 ago 4 11:01 [dirname]

gitlab-rails:
drwxr-xr-x 11 git root 4096 ago 4 14:19 .
drwxr-xr-x 24 root root 4096 ago 4 14:19 …

drwxr-x–x 11 git gitlab-www 11 ago 4 11:18 shared
drwx------ 45 git git 45 ago 4 11:19 uploads

gitlab-rails/shared:
drwxr-x–x 11 git gitlab-www 11 ago 4 11:18 .
drwxr-xr-x 11 git root 4096 ago 4 14:19 …
drwx------ 24 git root 24 ago 4 11:18 artifacts
drwx------ 3 git git 3 ago 4 11:18 cache
drwx------ 2 git root 2 ago 4 11:18 dependency_proxy
drwx------ 2 git root 2 ago 4 11:18 external-diffs
drwx------ 2 git root 2 ago 4 11:18 lfs-objects
drwx------ 2 git root 2 ago 4 11:18 packages
drwxr-x— 19 git gitlab-www 19 ago 4 11:18 pages
drwx------ 2 git root 2 ago 4 11:18 terraform_state
drwx------ 4 git git 4 ago 4 11:18 tmp

gitlab-rails/uploads:

drwxr-xr-x 3 git git 3 ago 4 11:19 [dirname]
drwxr-xr-x 2 git git 2 ago 4 11:19 [dirname]
drwxr-xr-x 2 git git 2 ago 4 11:19 [dirname]

gitlab-ci:
drwxr-xr-x 4 git root 4096 ago 4 11:41 .
drwxr-xr-x 24 root root 4096 ago 4 14:19 …
drwx------ 36 git git 36 ago 4 11:20 builds

gitlab-ci/builds:

drwxr-xr-x 4 git git 4 ago 4 11:20 2016_01
drwxr-xr-x 3 git git 3 ago 4 11:20 2016_02
drwxr-xr-x 3 git git 3 ago 4 11:20 2017_03

Hope this helps

Thank you for your explanation and sorry for my late reply. I still cannot find what was wrong on my end. I compared the 2 Gitlab instances and the permissions were indeed the same.

Strangely the error disappeared after running another upgrade of gitlab-ce today. Very weird. I didn’t touch the configuration at all.

I just had to run: gitlab-ctl reconfigure since gitlab-ce was complaining, during unpacking/upgrading, that the JSON configuration was invalid (no details given).