Gitaly is not starting under Ubnuntu/VirtualBox

Hi,

I’m trying to run GitLab CE under Ubuntu 24.04, which is running on VirtualBox with Windows as a host. My goal is to familiarise myself with GitLab configuration before I deploy it to the actual hardware, which I’m waiting for to arrive.

I have installed GitLab using the standard Linux installation procedure, and I can log in to it via the web interface, manage users, e.t.c. However, when I tried to create a new repo, I got a 503 error. I started to dig deeper and found that Gitaly is not starting.

The Gitaly log says the following:

{"level":"info","msg":"spawning a process","time":"2025-07-31T02:19:02Z","wrapper":20593}
{"gitaly":20603,"level":"info","msg":"monitoring gitaly","time":"2025-07-31T02:19:02Z","wrapper":20593}
time="2025-07-31T02:19:02Z" level=info msg="Starting Gitaly" version="Gitaly, version 13.0.6"
time="2025-07-31T02:19:02Z" level=warning msg="git path not configured. Using default path resolution" resolvedPath=/opt/gitlab/embedded/bin/git
time="2025-07-31T02:19:02Z" level=info msg="clearing disk cache object folder" path=/var/opt/gitlab/git-data/repositories
time="2025-07-31T02:19:02Z" level=info msg="moving disk cache object folder to /var/opt/gitlab/git-data/repositories/+gitaly/tmp/diskcache007544489" path=/var/opt/gitlab/git-data/repositories
time="2025-07-31T02:19:02Z" level=info msg="disk cache object folder doesn't exist, no need to remove" path=/var/opt/gitlab/git-data/repositories
time="2025-07-31T02:19:02Z" level=info msg="Starting file walker for /var/opt/gitlab/git-data/repositories/+gitaly/cache" path=/var/opt/gitlab/git-data/repositories/+gitaly/cache
time="2025-07-31T02:19:02Z" level=info msg="Starting file walker for /var/opt/gitlab/git-data/repositories/+gitaly/state" path=/var/opt/gitlab/git-data/repositories/+gitaly/state
time="2025-07-31T02:19:02Z" level=info msg="cleared all cache object files in /var/opt/gitlab/git-data/repositories/+gitaly/tmp/diskcache007544489 after 51.488µs" path=/var/opt/gitlab/git-data/repositories
{"error":"unable to start the bootstrap: can't create new listener: listen unix /tmp/gitaly.socket: operation not supported","level":"error","msg":"shutting down","time":"2025-07-31T02:19:02.692Z"}
{"gitaly":20603,"level":"warning","msg":"forwarding signal","signal":17,"time":"2025-07-31T02:19:02Z","wrapper":20593}
{"error":"os: process already finished","gitaly":20603,"level":"error","msg":"can't forward the signal","signal":17,"time":"2025-07-31T02:19:02Z","wrapper":20593}

This block of messages repeats every few seconds.
It looks like the main problem is this one:

unable to start the bootstrap: can't create new listener: listen unix /tmp/gitaly.socket: operation not supported

As you can see, I tried to specify /tmp as a directory for the socket file, but it didn’t help.
I’ve tried to search for solutions to this problem, but without success.
Please help me solve this problem.
Thanks!

I have no (relevant, we used it 10+ years ago for testing, but I don’t even think we had GitLab back then, and if we did, I didn’t administrate it) experience with VirtualBox, but we have gitaly running on ubuntu in an LXC guest (on top of another ubuntu), so it is possible to run it under some kind of virtualisation.

What does an ls -la /tmp on the server say?
(We run gitaly on separate servers so our gitaly socket is on TCP, so I don’t have anywhere to look for at configuration like yours that work)

I would also check your gitlab configuration - what exactly did you configure in /etc/gitlab/gitlab.rb? The reason I ask is that on my systems the socket file is here:

root@gitlab:~# find / -iname gitaly.socket
/var/opt/gitlab/gitaly/gitaly.socket

That varies from yours being placed in /tmp, which seems to be due to incorrect configuration.

As I read the question, that placement is intentional and an attempt to get around the mentioned problem.

I don’t think /tmp is a good place for a socket though, but in this case I don’t think the actual placement matters, the problem and it’s solution is probably the same. - Although if /tmp is a temporary filesystem (one of the reasons it’s a bad place for sockets) that might open another can of worms.

It might also be interesting to know where OP’s gitlab would place the socket if there was no configuration of it, and what the permissions of that place is.

Good point, didn’t notice that. Looking at permissions:

srwxr-xr-x.  1 git  git     0 Jul 31 06:00 gitaly.socket

if /tmp has been mounted separately, and with noexec mount option, it could possibly be a reason for failure. Assuming the execute bit is required on the socket file as shown on my permissions above. I don’t specify the location, and thus it’s in the location as shown. This is the same on all my other servers as well be it RHEL-based or Debian-based.

Either way, a basic Gitlab install with only the external_url configured should be enough to check/test. Why it’s failing could potentially suggest something wrong with the way the partitions have been used, and or VirtualBox doing something wrong. I know when people have attempted to use the VirtualBox plugins to mount a disk/share locally with the machine it’s running on can have adverse affects due to the permissions. But we don’t know if that is the case, and/or how the VM/server has been configured.

I haven’t used VirtualBox for years either, purely because it was always problematic. Natively under Linux you can use KVM and Virtual Machine Manager. Or there is also VMware Workstation Pro free for personal use now as well which is infinitely better than VirtualBox but that’s just my opinion. I personally use KVM only.

I just discovered that we actually have a way to bring up a server that has a local gitaly (that is based on running it in a qemu container, yet another kind of virtualisation, showing once more that gitaly and virtualisation is not a problem), but that reveals the same location and permissions as @iwalker has already shown.

The stuff he writes about /tmp being mounted separately is the stuff I was moving towards talking about it being a temporary filesystem, but as he indicates: the problems start long before.

@grove @iwalker Thank you for your replies!
Indeed, /tmp is an attempt to move the socket location to another directory to see if it changes anything. At some point, it looked to me like GitLab config changes are not affecting Gitaly anyhow, and I decided to make a noticeable change that has to appear in the log if Gitaly reads what I pass in the config. I had /var/opt/gitlab/gitaly/gitaly.socket as a socket path initially, and the error was the same. It feels like there is something wrong in the config, but I’m not sure where. I also tried to skip the socket config and just use this:

gitaly['enable'] = true
gitaly['dir'] = "/var/opt/gitlab/gitaly"
gitaly['log_directory'] = "/var/log/gitlab/gitaly"
gitaly['bin_path'] = "/opt/gitlab/embedded/bin/gitaly"
gitaly['env_directory'] = "/opt/gitlab/etc/gitaly/env"
gitaly['configuration'] = {
  listen_addr: '0.0.0.0:8075'
}                                                                                                                                                                                                                
git_data_dirs({
  'default' => {
    'path' => '/var/opt/gitlab/git-data'
  }
})
gitaly['listen_addr'] = "localhost:8075"

But this leads to another problem:

{"error":"open /var/opt/gitlab/gitaly/gitaly.pid: no such file or directory","level":"error","msg":"find gitaly","time":"2025-07-31T11:22:31Z","wrapper":21491}

I would be happy to just have a TCP connection, but I’m not sure what I’m missing with it and why Gitaly is trying to open pid file instead of creating it.
Talking about VirtualBox, I’m not a fan of it, either, and previously I had a very mixed experience with it, but with this Ubuntu installation, it works pretty smoothly, so I decided to continue with it for now. I may migrate to Docker if I see that VirtualBox is a problem, but for now, it seems that I just messed up the config somehow.

I’ve tried to keep just this line

external_url 'http://mysrv'

In this case, Gitaly shows the following error:

{"error":"open /var/opt/gitlab/gitaly/gitaly.pid: no such file or directory","level":"error","msg":"find gitaly","time":"2025-07-31T11:31:42Z","wrapper":32295}

roman@mysrv:~$ sudo ls -la /var/opt/gitlab/gitaly
total 20
drwx------  3 git  root 4096 Jul 31 11:29 .
drwxr-xr-x 21 root root 4096 Jul 31 11:29 ..
-rw-r--r--  1 root root   23 Jul 30 04:43 VERSION
-rw-r-----  1 root git   819 Jul 31 11:29 config.toml
drwx------  2 git  root 4096 Jul 31 11:32 internal_sockets

And just in case this is useful, /tmp is not mounted separately.
Here is its content and permissions:

roman@mysrv:~$ ls -la /tmp
total 52
drwxrwxrwt 12 root root 4096 Jul 31 11:33 .
drwxr-xr-x 23 root root 4096 Jul 30 04:22 ..
drwxrwxrwt  2 root root 4096 Jul 31 11:05 .ICE-unix
drwxrwxrwt  2 root root 4096 Jul 31 11:05 .X11-unix
drwxrwxrwt  2 root root 4096 Jul 31 11:05 .XIM-unix
drwxrwxrwt  2 root root 4096 Jul 31 11:05 .font-unix
lrwxrwxrwx  1 git  git    67 Jul 31 11:12 gitaly-linguist-path032356837 -> /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/github-linguist-7.9.0
drwx------  2 git  git  4096 Jul 31 11:10 prometheus-mmap20250731-6288-17335a5
drwx------  2 root root 4096 Jul 31 11:05 snap-private-tmp
drwx------  3 root root 4096 Jul 31 11:05 systemd-private-b2795f4b5cf24489a2c74b1e5905a6fd-apache2.service-ohf1YW
drwx------  3 root root 4096 Jul 31 11:05 systemd-private-b2795f4b5cf24489a2c74b1e5905a6fd-systemd-logind.service-gSevx8
drwx------  3 root root 4096 Jul 31 11:05 systemd-private-b2795f4b5cf24489a2c74b1e5905a6fd-systemd-resolved.service-5DrNfQ
drwx------  3 root root 4096 Jul 31 11:05 systemd-private-b2795f4b5cf24489a2c74b1e5905a6fd-systemd-timesyncd.service-zyXC5O

What am I missing?

I would be tempted after the changes, to get back to a clean system. So, remove the gitlab-ce package:

apt-get remove gitlab-ce --purge

this should get rid of the package from apt, and ensure config files are remove. Then ensure that /etc/gitlab, /opt/gitlab, /var/opt/gitlab and /var/log/gitlab are deleted.

Then follow the gitlab install instructions again to get gitlab-ce installed, and just change the external_url - which is explained in the install docs anyway, don’t change anything else for now. When Gitlab is running after the install, at this point verify it’s status:

1. Is webgui working? Can you visit various pages, including the admin backend.
2. Run the command gitlab-ctl status to check all services are running.
3. If the above command shows something not running, do gitlab-ctl reconfigure to try and fix that. Then do gitlab-ctl restart afterwards to check again if all services are running afterwards with the status command in point 2.

Sometimes you may need to do systemctl restart gitlab-runsvdir or reboot the system, and recheck if all services are running.

Once we get back to a basic install without any attempted fixes we can then easily debug it further as we should be more or less at a known constant.

This is exactly the config that I had at the very beginning. The only change I had essentially is external_url. This led to the first error that I’ve mentioned in the first message. After that I’ve started the experimentation with /tmp and TCP. But I’ll try it anyway. Thanks!

Yep, the thing is once some changes have been made, it’s difficult to figure out what next. If we’ve got a clean install which has only had the external_url changed, and the error still appears then we know it’s not config related in terms of Gitlab. At which point, we need to look at the way the VM is running, disk mounts, permissions etc, because this problem shouldn’t occur.

I’ve made plenty of installs over the years, with older and newer versions and never had this issue. I’ve even recently installed new VM’s and restored my production server to test upgrades, etc, and this also didn’t happen.

There’s obviously something wrong with the VM itself, or the environment, because otherwise gitlab would install and run immediately.

@iwalker full clean reinstall helped!
Not sure what I’ve changed last time, that definitely broke something
I have reinstalled everything with external_url and it works now.
Couple of things that I’ve learned:

1. I had to remove Apache2 installed on Ubuntu by default to make port 80 available
2. I tried to use port 8080 instead, but apparently it is used by Ruby

After removing Apache and setting external_url to http://192.168.1.110, everything started to work, and I managed to create a new repo, clone it and push into it.
Now, when I know the bare minimum of configuration needed, the installation looks extremely simple
Thanks for your help!

Yep, Gitlab has nginx on port 80 by default, port 443 as well when SSL is configured. Port 8080 is used by puma, which nginx redirects all http/https traffic to.