Directories in /var/opt/gitlab/ on clustered gitlab vmware VMs missing after storage failure

Hi folks,
I had no part in building our clustered Gitlab servers but now I’m tasked with figuring out why they’re broken after a storage array failure where the host esxi servers lost their storage and consequently the guest VMs were not properly shut down.
Storage is now fixed and the VMs are now started. However, NGINX, Postgresql, and gitlab-workhorse will not start because their corresponding directories are missing from /var/opt/gitlab/.
Here are the exact errors:
2021-12-07_21:14:45.75025 ./run: line 9: cd: /var/opt/gitlab/gitlab-workhorse: No such file or directory
2021-12-07_21:15:09.02896 postgres cannot access the server configuration file “/var/opt/gitlab/postgresql/data/postgresql.conf”: No such file or directory
2021-12-07_21:15:35.56285 2021/12/07 16:15:35 [emerg] 26923#0: open() “/var/opt/gitlab/nginx/conf/nginx.conf” failed (2: No such file or directory)

I find it an odd coincidence that both servers are missing the exact same directories and are both unable to start. I see no evidence that they share storage between each other.

Servers are running the following:
Oracle Linux Server 7.3
gitlab-ce 8.14.1

Any ideas would be appreciated. I’m finding no existing documentation regarding this issue and my ignorance about gitlab is making it difficult to come up with next steps.

Thanks!

You would need to restore those directories from a backup. Considering that the postgresql data directory is also missing means you have no database and that means gitlab will no longer work.

Thank you iwalker. We’re working that angle now.

Turns out we’re using drbd and a shared disk. Because of the storage outage drbd became split-brained and I needed to force one server to become primary and the other secondary. After that I was able to mount the shared drbd resource at /var/opt/gitlab using the standard linux mount command.

1 Like