GitLab Upgrade 16.11 to 17.3.5 - Error

Problem to solve

I am trying to upgrade GitLab from 16.11–>17.3.5–>17.4.2. When upgrade to 17.3.5 got below error.

500: We’re sorry, something went wrong on our end

From Gitaly logs as below
{“level”:“warning”,“msg”:“The current configurations will cause Gitaly to fail to start up in future versions. Please run ‘gitaly configuration validate \u003c /var/opt/gitlab/gitaly/config.toml’ and fix the errors that are printed.”,“pid”:46135,“time”:“2024-10-11T07:59:34.981Z”}

unclean Gitaly shutdown: creating Git command factory: setting up Git execution environment: constructing Git environment: checking bundled Git binary “gitaly-git-v2.45”: permission denied

Unable to access GitLab after upgrade

hi @ravikasturi9 first of all you downgrade your gitlab to 16.11 after that use gitlab update tool

Since his path 16.11 → 17.3.5 → 17.4.2 is exactly as the update path tool shows, the above post doesn’t help at all whatsoever.

@ravikasturi9 please try doing this after the upgrade:

gitlab-ctl reconfigure
gitlab-ctl restart
systemctl restart gitlab-runsvdir

if anything throws up errors during the gitlab-ctl reconfigure address whatever is incorrectly configured in /etc/gitlab/gitlab.rb.

Thanks for reply. I tried those steps already. Still got same error as stated in error msgs

Can you try running this command:

root@gitlab:~# /opt/gitlab/embedded/bin/gitaly configuration validate /var/opt/gitlab/gitaly/config.toml 

also are you using some specific gitaly configuration in /etc/gitlab/gitlab.rb that may be incorrect? Since the error would suggest something is being used that is either no longer or shortly no longer going to be supported. Check against deprecations mentioned here: Deprecations and removals by version | GitLab there’s plenty mentioned related to gitaly.

Hi, Looks you are right. I try to run configuration validate but not giving any such response. Hung there.

Below is the current configuration. Any suggestion by looking at? Not using any custom attributes apart from this.

Gitaly configuration file

This file is managed by gitlab-ctl. Manual changes will be

erased! To change the contents below, edit /etc/gitlab/gitlab.rb

and run:

sudo gitlab-ctl reconfigure

runtime_dir = “/var/opt/gitlab/gitaly/run”
socket_path = “/var/opt/gitlab/gitaly/gitaly.socket”
prometheus_listen_addr = “localhost:9236”
bin_dir = “/opt/gitlab/embedded/bin”

[logging]
dir = “/var/log/gitlab/gitaly”
format = “json”

[git]
use_bundled_binaries = true
bin_path = “/opt/gitlab/embedded/bin/git”
ignore_gitconfig = true

[[storage]]
name = “default”
path = “/var/opt/gitlab/git-data/repositories”

[gitlab]
url = “http+unix://%2Fvar%2Fopt%2Fgitlab%2Fgitlab-workhorse%2Fsockets%2Fsocket”
relative_url_root = “”

[gitlab-shell]
dir = “/opt/gitlab/embedded/service/gitlab-shell”

The above config you quoted is from the gitaly config.toml - I asked you to check /etc/gitlab/gitlab.rb since all changes are made in that main gitlab.rb file. Your problem is most likely as I said already, incorrect configuration of gitlab.rb due to deprecations.

@iwalker, we are not using any gitaly related in gitlab.rb file

I guess you’ll have to go through all the log files under /var/log/gitlab until you find something more that hints at the problem. To me it looks like Gitlab is incorrectly configured or using something that has been deprecated between versions. But you say that’s not the case.

Check your configuration and check your logs, it’s impossible to help otherwise.

Went through the documentation once again. From the logs I see only to run configuration validation error related.Here not using gitaly much for runners…etc. Basic purpose.

Surpise to see errors.

@iwalker , Found something intresting, may help if you came across such condition.

When ever I upgrade to v17.3.5, I see some directories are created under /var/opt/gitlab/gitaly/run/

It is unable to get latest GitLab version files. The directories are pointing to older version of git.git-http.

ln -sf /opt/gitlab/embedded/bin/gitaly-git-remote-http-v2.45 git-remote-https
ln -sf /opt/gitlab/embedded/bin/gitaly-git-v2.45 git-upload-archive
ln -sf /opt/gitlab/embedded/bin/gitaly-git-v2.45 git-upload-pack

If I create those link files manually with correct version, then gitlab is coming up. After Gitaly/Gitlab is up, if I restart, again it is collapsing with same error. Any idea, is there any specific migration commands to run? Or is it Bug in upgrade?

checking bundled Git binary “gitaly-git-http-backend-v2.45”: permission denied

1 Like

Strange, not seen anything like that, but this is how my directory looks:

root@gitlab:/var/opt/gitlab/gitaly/run# ls -lhR
.:
total 4.0K
drwx------. 5 git git 4.0K Oct  9 22:11 gitaly-128093

./gitaly-128093:
total 136M
-r-x------. 1 git git 2.5M Oct  9 22:11 gitaly-git-http-backend-v2.46
-r-x------. 1 git git 2.6M Oct  9 22:11 gitaly-git-remote-http-v2.46
-r-x------. 1 git git 4.2M Oct  9 22:11 gitaly-git-v2.46
-r-x------. 1 git git  18M Oct  9 22:11 gitaly-gpg
-r-x------. 1 git git  38M Oct  9 22:11 gitaly-hooks
-r-x------. 1 git git  38M Oct  9 22:11 gitaly-lfs-smudge
-r-x------. 1 git git  34M Oct  9 22:11 gitaly-ssh
drwx------. 2 git git 4.0K Oct  9 22:11 git-exec-2328860727.d
drwx------. 2 git git  108 Oct  9 22:11 hooks-2143991876.d
drwx------. 2 git git   20 Oct  9 22:11 sock.d

./gitaly-128093/git-exec-2328860727.d:
total 0
lrwxrwxrwx. 1 git git 57 Oct  9 22:11 git -> /var/opt/gitlab/gitaly/run/gitaly-128093/gitaly-git-v2.46
lrwxrwxrwx. 1 git git 70 Oct  9 22:11 git-http-backend -> /var/opt/gitlab/gitaly/run/gitaly-128093/gitaly-git-http-backend-v2.46
lrwxrwxrwx. 1 git git 57 Oct  9 22:11 git-receive-pack -> /var/opt/gitlab/gitaly/run/gitaly-128093/gitaly-git-v2.46
lrwxrwxrwx. 1 git git 69 Oct  9 22:11 git-remote-ftp -> /var/opt/gitlab/gitaly/run/gitaly-128093/gitaly-git-remote-http-v2.46
lrwxrwxrwx. 1 git git 69 Oct  9 22:11 git-remote-ftps -> /var/opt/gitlab/gitaly/run/gitaly-128093/gitaly-git-remote-http-v2.46
lrwxrwxrwx. 1 git git 69 Oct  9 22:11 git-remote-http -> /var/opt/gitlab/gitaly/run/gitaly-128093/gitaly-git-remote-http-v2.46
lrwxrwxrwx. 1 git git 69 Oct  9 22:11 git-remote-https -> /var/opt/gitlab/gitaly/run/gitaly-128093/gitaly-git-remote-http-v2.46
lrwxrwxrwx. 1 git git 57 Oct  9 22:11 git-upload-archive -> /var/opt/gitlab/gitaly/run/gitaly-128093/gitaly-git-v2.46
lrwxrwxrwx. 1 git git 57 Oct  9 22:11 git-upload-pack -> /var/opt/gitlab/gitaly/run/gitaly-128093/gitaly-git-v2.46

./gitaly-128093/hooks-2143991876.d:
total 0
lrwxrwxrwx. 1 git git 53 Oct  9 22:11 post-receive -> /var/opt/gitlab/gitaly/run/gitaly-128093/gitaly-hooks
lrwxrwxrwx. 1 git git 53 Oct  9 22:11 pre-receive -> /var/opt/gitlab/gitaly/run/gitaly-128093/gitaly-hooks
lrwxrwxrwx. 1 git git 53 Oct  9 22:11 proc-receive -> /var/opt/gitlab/gitaly/run/gitaly-128093/gitaly-hooks
lrwxrwxrwx. 1 git git 53 Oct  9 22:11 reference-transaction -> /var/opt/gitlab/gitaly/run/gitaly-128093/gitaly-hooks
lrwxrwxrwx. 1 git git 53 Oct  9 22:11 update -> /var/opt/gitlab/gitaly/run/gitaly-128093/gitaly-hooks

./gitaly-128093/sock.d:
total 0
srwxr-xr-x. 1 git git 0 Oct  9 22:11 intern

when I stop Gitlab, that directory is then empty.

root@gitlab:/var/opt/gitlab/gitaly/run# ls -lhR
.:
total 0

When I start again, everything is populated automatically.

So something is wrong with your installation - do you use NFS by any chance for your storage? Because that was deprecated a long time back, so can only assume if files are not being created perhaps it is because of this or some other permissions problem.

The above however is for Gitlab 17.4.2.

No, not using any NFS. Strange behaviour. Looks the link files are not generating when GITALY starts.

between, possible to share how to export and import data. Work around is to remove current version of GitLab by exporting data and import after upgrade to latest.

There are posts already on the forum about how to export/import data from Gitlab. Gitlab documentation also has procedures.

@iwalker Thanks. Will go through that. We can close this.

Hi,

i also had issues with the GitLab upgrade from 17.3.6 to 17.5.1. files under /var/opt/gitlab/gitaly/run was missing. My first thought was my CIS Level 1 Compliant Installation of AlmaLinux, it puts an noexec on /var and some others directorys that should not host executables, and i was right, after removing noexec on /var, gitaly came up, no reconfigure required.

Please do not place executables or linked executables in /var, from CIS Level 1 or Level 2 Compliance standpoint, this is a no go.

I moved /var/opt/gitlab to a different partition without noexec, everything works fine now.

Best regards.

3 Likes

This also happens on our Docker hosts too, from 17.3.6-ee to 17.4.3-ee

The workarounds unfortunately don’t help as the containers are reset to Gitlab’s default after restarting.

unclean Gitaly shutdown: creating Git command factory: setting up Git execution environment: constructing Git environment: Checking bundled Git binary "gitaly-git-v2.46": permission denied

What does this actually mean?

Like OP we have minimal setup, the only manually edited gitlab.rb changes are for the host url, SMTP and LDAPS. None of which have any breaking changes in 17.x

There is a path mount (host) /GitLabs/data to /var/opt/gitlabs (container) if that makes any difference? It has write privileges so I assume not

Thanks :slight_smile:

Gave up trying to fix it, initiated a restore from backup (Veeam) and it works fine pre-upgrade.

@ravikasturi9 Sorry to bother but are you using an Offline installation? The only difference we have between the usual install of GItlabs is that we have no internet connected on our host. It could be causing the issue if it’s trying to download a package from the internet after starting.

Nope, if it has CIS issues, it cannot run 16.11 version also. Looks like there are some major changes from v16 to v17 which required to read and fix before go to 17.