I am trying to upgrade GitLab from 16.11–>17.3.5–>17.4.2. When upgrade to 17.3.5 got below error.
500: We’re sorry, something went wrong on our end
From Gitaly logs as below
{“level”:“warning”,“msg”:“The current configurations will cause Gitaly to fail to start up in future versions. Please run ‘gitaly configuration validate \u003c /var/opt/gitlab/gitaly/config.toml’ and fix the errors that are printed.”,“pid”:46135,“time”:“2024-10-11T07:59:34.981Z”}
also are you using some specific gitaly configuration in /etc/gitlab/gitlab.rb that may be incorrect? Since the error would suggest something is being used that is either no longer or shortly no longer going to be supported. Check against deprecations mentioned here: Deprecations and removals by version | GitLab there’s plenty mentioned related to gitaly.
The above config you quoted is from the gitaly config.toml - I asked you to check /etc/gitlab/gitlab.rb since all changes are made in that main gitlab.rb file. Your problem is most likely as I said already, incorrect configuration of gitlab.rb due to deprecations.
I guess you’ll have to go through all the log files under /var/log/gitlab until you find something more that hints at the problem. To me it looks like Gitlab is incorrectly configured or using something that has been deprecated between versions. But you say that’s not the case.
Check your configuration and check your logs, it’s impossible to help otherwise.
Went through the documentation once again. From the logs I see only to run configuration validation error related.Here not using gitaly much for runners…etc. Basic purpose.
If I create those link files manually with correct version, then gitlab is coming up. After Gitaly/Gitlab is up, if I restart, again it is collapsing with same error. Any idea, is there any specific migration commands to run? Or is it Bug in upgrade?
Strange, not seen anything like that, but this is how my directory looks:
root@gitlab:/var/opt/gitlab/gitaly/run# ls -lhR
.:
total 4.0K
drwx------. 5 git git 4.0K Oct 9 22:11 gitaly-128093
./gitaly-128093:
total 136M
-r-x------. 1 git git 2.5M Oct 9 22:11 gitaly-git-http-backend-v2.46
-r-x------. 1 git git 2.6M Oct 9 22:11 gitaly-git-remote-http-v2.46
-r-x------. 1 git git 4.2M Oct 9 22:11 gitaly-git-v2.46
-r-x------. 1 git git 18M Oct 9 22:11 gitaly-gpg
-r-x------. 1 git git 38M Oct 9 22:11 gitaly-hooks
-r-x------. 1 git git 38M Oct 9 22:11 gitaly-lfs-smudge
-r-x------. 1 git git 34M Oct 9 22:11 gitaly-ssh
drwx------. 2 git git 4.0K Oct 9 22:11 git-exec-2328860727.d
drwx------. 2 git git 108 Oct 9 22:11 hooks-2143991876.d
drwx------. 2 git git 20 Oct 9 22:11 sock.d
./gitaly-128093/git-exec-2328860727.d:
total 0
lrwxrwxrwx. 1 git git 57 Oct 9 22:11 git -> /var/opt/gitlab/gitaly/run/gitaly-128093/gitaly-git-v2.46
lrwxrwxrwx. 1 git git 70 Oct 9 22:11 git-http-backend -> /var/opt/gitlab/gitaly/run/gitaly-128093/gitaly-git-http-backend-v2.46
lrwxrwxrwx. 1 git git 57 Oct 9 22:11 git-receive-pack -> /var/opt/gitlab/gitaly/run/gitaly-128093/gitaly-git-v2.46
lrwxrwxrwx. 1 git git 69 Oct 9 22:11 git-remote-ftp -> /var/opt/gitlab/gitaly/run/gitaly-128093/gitaly-git-remote-http-v2.46
lrwxrwxrwx. 1 git git 69 Oct 9 22:11 git-remote-ftps -> /var/opt/gitlab/gitaly/run/gitaly-128093/gitaly-git-remote-http-v2.46
lrwxrwxrwx. 1 git git 69 Oct 9 22:11 git-remote-http -> /var/opt/gitlab/gitaly/run/gitaly-128093/gitaly-git-remote-http-v2.46
lrwxrwxrwx. 1 git git 69 Oct 9 22:11 git-remote-https -> /var/opt/gitlab/gitaly/run/gitaly-128093/gitaly-git-remote-http-v2.46
lrwxrwxrwx. 1 git git 57 Oct 9 22:11 git-upload-archive -> /var/opt/gitlab/gitaly/run/gitaly-128093/gitaly-git-v2.46
lrwxrwxrwx. 1 git git 57 Oct 9 22:11 git-upload-pack -> /var/opt/gitlab/gitaly/run/gitaly-128093/gitaly-git-v2.46
./gitaly-128093/hooks-2143991876.d:
total 0
lrwxrwxrwx. 1 git git 53 Oct 9 22:11 post-receive -> /var/opt/gitlab/gitaly/run/gitaly-128093/gitaly-hooks
lrwxrwxrwx. 1 git git 53 Oct 9 22:11 pre-receive -> /var/opt/gitlab/gitaly/run/gitaly-128093/gitaly-hooks
lrwxrwxrwx. 1 git git 53 Oct 9 22:11 proc-receive -> /var/opt/gitlab/gitaly/run/gitaly-128093/gitaly-hooks
lrwxrwxrwx. 1 git git 53 Oct 9 22:11 reference-transaction -> /var/opt/gitlab/gitaly/run/gitaly-128093/gitaly-hooks
lrwxrwxrwx. 1 git git 53 Oct 9 22:11 update -> /var/opt/gitlab/gitaly/run/gitaly-128093/gitaly-hooks
./gitaly-128093/sock.d:
total 0
srwxr-xr-x. 1 git git 0 Oct 9 22:11 intern
when I stop Gitlab, that directory is then empty.
root@gitlab:/var/opt/gitlab/gitaly/run# ls -lhR
.:
total 0
When I start again, everything is populated automatically.
So something is wrong with your installation - do you use NFS by any chance for your storage? Because that was deprecated a long time back, so can only assume if files are not being created perhaps it is because of this or some other permissions problem.
between, possible to share how to export and import data. Work around is to remove current version of GitLab by exporting data and import after upgrade to latest.
i also had issues with the GitLab upgrade from 17.3.6 to 17.5.1. files under /var/opt/gitlab/gitaly/run was missing. My first thought was my CIS Level 1 Compliant Installation of AlmaLinux, it puts an noexec on /var and some others directorys that should not host executables, and i was right, after removing noexec on /var, gitaly came up, no reconfigure required.
Please do not place executables or linked executables in /var, from CIS Level 1 or Level 2 Compliance standpoint, this is a no go.
I moved /var/opt/gitlab to a different partition without noexec, everything works fine now.
Like OP we have minimal setup, the only manually edited gitlab.rb changes are for the host url, SMTP and LDAPS. None of which have any breaking changes in 17.x
There is a path mount (host) /GitLabs/data to /var/opt/gitlabs (container) if that makes any difference? It has write privileges so I assume not
Gave up trying to fix it, initiated a restore from backup (Veeam) and it works fine pre-upgrade.
@ravikasturi9 Sorry to bother but are you using an Offline installation? The only difference we have between the usual install of GItlabs is that we have no internet connected on our host. It could be causing the issue if it’s trying to download a package from the internet after starting.
Nope, if it has CIS issues, it cannot run 16.11 version also. Looks like there are some major changes from v16 to v17 which required to read and fix before go to 17.