GitLab 9.0.4: Failed to connect to internal API + 502 error

When attempting to reach the web interface I get an instant 502 timeout error. No waiting, auto-502.

Background: I upgraded Ubuntu Server from 12.04 to 16.04, and in the process upgraded Gitlab from 8.* (believe it was Omnibus) to gitlab-ce 9.0.4. Everything was running fine prior to upgrade. During upgrade I had a few issues with file permissions, got those fixed, but now I am stuck.

I ran sudo gitlab-rake gitlab:check SANITIZE=true and everything passed except:

Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Check GitLab API access: FAILED: Failed to connect to internal API
gitlab-shell self-check failed
  Try fixing it:
  Make sure GitLab is running;
  Check the gitlab-shell configuration file:
  sudo -u git -H editor /opt/gitlab/embedded/service/gitlab-shell/config.yml
  Please fix the error above and rerun the checks.

My environment via sudo gitlab-rake gitlab:env:info SANITIZE=true:

Environment
System information
System:		Ubuntu 16.04
Current User:	git
Using RVM:	no
Ruby Version:	2.3.3p222
Gem Version:	2.6.6
Bundler Version:1.13.7
Rake Version:	10.5.0
Redis Version:	3.2.5
Git Version:	2.10.2
Sidekiq Version:4.2.7

GitLab information
Version:	9.0.4
Revision:	63e052a
Directory:	/opt/gitlab/embedded/service/gitlab-rails
DB Adapter:	postgresql
URL:		https://*****:4040
HTTP Clone URL:	https://******:4040/some-group/some-project.git
SSH Clone URL:	git@*******:some-group/some-project.git
Using LDAP:	no
Using Omniauth:	no

GitLab Shell
Version:	5.0.0
Repository storage paths:
- default: 	/media/***/gitlab/repositories
Hooks:		/opt/gitlab/embedded/service/gitlab-shell/hooks/
Git:		/opt/gitlab/embedded/bin/git

I censored the URL, but it is web-accessible and uses a self-signed certificate.

Finally, here is the modified parts of my gitlab.rb:

external_url 'https://*****:4040'
gitlab_rails['backup_path'] = "/media/****/gitlab/backup"
git_data_dir "/media/*****/gitlab"

I already ran sudo gitlab-ctl reconfigure, sudo gitlab-ctl upgrade and sudo gitlab-ctl restart. Restarted several times. Waited overnight to ensure all post-processing was complete. Nothing changed.

I also noticed that netstat -ltn | grep 8080 returns nothing, implying unicorn is not running, but 24335 ? 00:00:00 gitlab-unicorn- is there in ps -A | grep unicorn.

Suggestions?

Edit: I tried purging every gitlab file I could find, reinstalling gitlab-ce omnibus package, and manually reconfiguring. Same error, nothing changed! I like Gitlab quite a bit, but if I can’t get over this last issue I’ll have to look for alternatives.

FWIW, I ended up switching to Gogs which is written in Go. Faster, easier to configure, simpler code base, has the same features from Gitlab that I care about, supposedly less buggy to upgrade, and looks practically identical to Github. Can’t imagine I’ll ever come back to Gitlab, but you never know.