Gitlab fails to run after upgrade from 14.9 to 14.10

Installation on Ubuntu 18.04 LTS. Upgraded via package repo:

sudo apt update
sudo apt install gitlab-ce

Upgrade reported successful. I rebooted and ran:

sudo gitlab-ctl reconfigure

I see:

Running handlers:
There was an error running gitlab-ctl reconfigure:

directory[/var/opt/gitlab/gitaly/internal_sockets] (gitaly::enable line 63) had an error: Errno::ENOTEMPTY: Directory not empty @ dir_s_rmdir - /var/opt/gitlab/gitaly/internal_sockets

Attempt to login to to web interface results in 500 error Whoops, something went wrong on our end.

There are 5 socket files:

/var/opt/gitlab/gitaly/internal_sockets# file {.,}*
.:                            directory
..:                           directory
.nfs00000000a4f1b6b000000005: socket
.nfs00000000a4f1cbe100000011: socket
.nfs00000000a4ffe7a30000000f: socket
ruby.0:                       socket
ruby.1:                       socket

I am guessing the issue is related to this:

but I don’t even have gitaly[‘internal_socket_dir’] configured so I am not sure what to do to allow to resolve the error. I couldn’t keep the service offline any longer so I reverted to a snapshot of the VM. What do I need to do to avoid this error when I attempt the upgrade again?

I upgraded two of mine last night from 14.9.3 to 14.10.0, prior to upgrade internal_sockets existed, after upgrade, it was gone. I do not have any issues with my version not working though, all works fine.

gitlab-ctl status
run: gitaly: (pid 1473721) 86954s; run: log: (pid 1441) 2060096s
run: gitlab-kas: (pid 1472085) 86961s; run: log: (pid 1427) 2060096s
run: gitlab-workhorse: (pid 1472208) 86961s; run: log: (pid 1429) 2060096s
run: logrotate: (pid 1028681) 552s; run: log: (pid 1432) 2060096s
run: nginx: (pid 1472452) 86960s; run: log: (pid 1439) 2060096s
run: postgresql: (pid 1018572) 1044486s; run: log: (pid 1431) 2060096s
run: puma: (pid 1472551) 86959s; run: log: (pid 1450) 2060096s
run: redis: (pid 1018796) 1044485s; run: log: (pid 1430) 2060096s
run: sidekiq: (pid 1472632) 86959s; run: log: (pid 1433) 2060096s

and the directory contents:

ls -lha /var/opt/gitlab/gitaly/

total 28K
drwx------  3 git  root 4.0K Apr 24 17:12 .
drwxr-xr-x 22 root root 4.0K Apr 24 17:10 ..
-rw-r-----  1 root git   983 Apr 24 17:10 config.toml
-rw-------  1 git  git     7 Apr 24 17:12
srwxr-xr-x  1 git  git     0 Apr 13 15:13 gitaly.socket
-rw-r--r--  1 root root   63 Dec  7 08:25 RUBY_VERSION
drwx------  3 git  root 4.0K Apr 24 17:12 run
-rw-r--r--  1 root root   72 Apr 24 17:10 VERSION

so I think if that really is the reason for it, then you can completely delete internal_sockets altogether. Just out of interest, are you using NFS? The reason I ask is seeing those .nfs files and also because NFS is deprecated Using NFS with GitLab | GitLab

1 Like

Yeah my other instances upgraded fine as well and I saw the same behavior as you. This is the only one with an NFS mount so maybe that is interfering.

I will remove the NFS and try to upgrade again this weekend. I’ll post the results here and mark as resolved if it does the trick.



Migration from NFS to ESX datastore allowed the upgrade to complete without resulting errors.