Hi, recently I encountered quite a weird problem while updating GitLab and not following the right update instructions (updating from 11 to 12 without following the update instructions). I am using a self-managed instance, and the version my latest backup is on (and which I am trying to get working again) is 11.5.0. After updating wrongly, I noticed GitLab gave HTTP 500/502/503 codes, so I uninstalled GitLab, reinstalled the previous version, ran gitlab-ctl reconfigure and yeah, it does not work. It gives me the following error:
There was an error running gitlab-ctl reconfigure:
env_dir[/opt/gitlab/etc/gitaly] (gitaly::enable line 53) had an error: Errno::EISDIR: file[/opt/gitlab/etc/gitaly/env] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/package/resources/env_dir.rb line 30) had an error: Errno::EISDIR: Is a directory - read
When running gitlab-ctl status I see
run: alertmanager: (pid 9118) 2165s; run: log: (pid 13608) 2661s
down: gitaly: 1s, normally up, want up; run: log: (pid 13622) 2661s
down: gitlab-exporter: 1s, normally up, want up; run: log: (pid 13606) 2661s
run: gitlab-monitor: (pid 10026) 2127s; run: log: (pid 13638) 2661s
run: gitlab-workhorse: (pid 10036) 2127s; run: log: (pid 13632) 2661s
run: logrotate: (pid 10047) 2126s; run: log: (pid 13637) 2661s
run: node-exporter: (pid 10055) 2126s; run: log: (pid 13607) 2661s
run: postgres-exporter: (pid 10061) 2125s; run: log: (pid 13610) 2661s
run: postgresql: (pid 10100) 2124s; run: log: (pid 13625) 2661s
down: prometheus: 0s, normally up, want up; run: log: (pid 13634) 2661s
run: sidekiq: (pid 10474) 2102s; run: log: (pid 13609) 2661s
run: unicorn: (pid 10522) 2100s; run: log: (pid 13624) 2661s
And gitlab-ctl tail tells me that the following errors are occuring all the time:
==> /var/log/gitlab/prometheus/current <==
2020-04-14_08:31:38.92000 chpst: fatal: unable to run: /opt/gitlab/embedded/bin/prometheus: file does not exist
==> /var/log/gitlab/gitlab-exporter/current <==
2020-04-14_08:31:40.13748 chpst: fatal: unable to run: /opt/gitlab/embedded/bin/gitlab-exporter: file does not exist
==> /var/log/gitlab/gitaly/current <==
chpst: fatal: unable to run: /opt/gitlab/embedded/bin/gitaly-wrapper: file does not exist
So yeah, I am out of my depth here. Any help would be hugely appreciated.
One of the things that might have caused this is me installing JFrog Artifactory, which by default uses the ports 8081 and 8082 (which I have set GitLab to use), however I changed that port configuration. lsof tells me the following:
# lsof -i:8081 (GitLab)
# lsof -i:8082 (Sidekiq)
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
bundle 10474 git 17u IPv4 11405843 0t0 TCP localhost:8082 (LISTEN)
# lsof -i:8084 (Unicorn)
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
bundle 10543 git 18u IPv4 11406620 0t0 TCP localhost:8084 (LISTEN)
bundle 11471 git 18u IPv4 11406620 0t0 TCP localhost:8084 (LISTEN)
bundle 11474 git 18u IPv4 11406620 0t0 TCP localhost:8084 (LISTEN)
bundle 11477 git 18u IPv4 11406620 0t0 TCP localhost:8084 (LISTEN)
# lsof -i:9229 (Workhorse)
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
gitlab-wo 10036 git 5u IPv4 11397644 0t0 TCP localhost:9229 (LISTEN)
Thanks in advanced! If you need any more information please let me know.