Gitlab fails to run after upgrade from 14.9 to 14.10

Installation on Ubuntu 18.04 LTS. Upgraded via package repo:

sudo apt update
sudo apt install gitlab-ce

Upgrade reported successful. I rebooted and ran:

sudo gitlab-ctl reconfigure

I see:

Running handlers:
There was an error running gitlab-ctl reconfigure:

directory[/var/opt/gitlab/gitaly/internal_sockets] (gitaly::enable line 63) had an error: Errno::ENOTEMPTY: Directory not empty @ dir_s_rmdir - /var/opt/gitlab/gitaly/internal_sockets

Attempt to login to to web interface results in 500 error Whoops, something went wrong on our end.

There are 5 socket files:

/var/opt/gitlab/gitaly/internal_sockets# file {.,}*
.:                            directory
..:                           directory
.nfs00000000a4f1b6b000000005: socket
.nfs00000000a4f1cbe100000011: socket
.nfs00000000a4ffe7a30000000f: socket
ruby.0:                       socket
ruby.1:                       socket

I am guessing the issue is related to this:

but I don’t even have gitaly[‘internal_socket_dir’] configured so I am not sure what to do to allow to resolve the error. I couldn’t keep the service offline any longer so I reverted to a snapshot of the VM. What do I need to do to avoid this error when I attempt the upgrade again?

I upgraded two of mine last night from 14.9.3 to 14.10.0, prior to upgrade internal_sockets existed, after upgrade, it was gone. I do not have any issues with my version not working though, all works fine.

gitlab-ctl status
run: gitaly: (pid 1473721) 86954s; run: log: (pid 1441) 2060096s
run: gitlab-kas: (pid 1472085) 86961s; run: log: (pid 1427) 2060096s
run: gitlab-workhorse: (pid 1472208) 86961s; run: log: (pid 1429) 2060096s
run: logrotate: (pid 1028681) 552s; run: log: (pid 1432) 2060096s
run: nginx: (pid 1472452) 86960s; run: log: (pid 1439) 2060096s
run: postgresql: (pid 1018572) 1044486s; run: log: (pid 1431) 2060096s
run: puma: (pid 1472551) 86959s; run: log: (pid 1450) 2060096s
run: redis: (pid 1018796) 1044485s; run: log: (pid 1430) 2060096s
run: sidekiq: (pid 1472632) 86959s; run: log: (pid 1433) 2060096s

and the directory contents:

ls -lha /var/opt/gitlab/gitaly/

total 28K
drwx------  3 git  root 4.0K Apr 24 17:12 .
drwxr-xr-x 22 root root 4.0K Apr 24 17:10 ..
-rw-r-----  1 root git   983 Apr 24 17:10 config.toml
-rw-------  1 git  git     7 Apr 24 17:12 gitaly.pid
srwxr-xr-x  1 git  git     0 Apr 13 15:13 gitaly.socket
-rw-r--r--  1 root root   63 Dec  7 08:25 RUBY_VERSION
drwx------  3 git  root 4.0K Apr 24 17:12 run
-rw-r--r--  1 root root   72 Apr 24 17:10 VERSION

so I think if that really is the reason for it, then you can completely delete internal_sockets altogether. Just out of interest, are you using NFS? The reason I ask is seeing those .nfs files and also because NFS is deprecated Using NFS with GitLab | GitLab

1 Like

Yeah my other instances upgraded fine as well and I saw the same behavior as you. This is the only one with an NFS mount so maybe that is interfering.

I will remove the NFS and try to upgrade again this weekend. I’ll post the results here and mark as resolved if it does the trick.

Thanks!

2 Likes

Migration from NFS to ESX datastore allowed the upgrade to complete without resulting errors.

2 Likes