Error 500 after reboot and possible upgrades

How to Use GitLab
1

Hi all,

I went to connect to my GitLab box today and received the error:

500
Whoops, something went wrong on our end.

I headed into the logs and couldn’t find much, but after some searching I came across gitlab-rake gitlab:check.

Running this I found that the Postgres socket wasn’t where GitLab thought it should be (/var/opt/gitlab/postgresql/.s.PGSQL.5432). So I headed into the Postgres conf file and changed unix_socket_directories to point to the ‘correct’ location.
Postgres then complained that it didn’t have permission to that directory. I took a look and the perms were set to gitlab-sql:gitlab-sql, so I changed them to postgres:postgres and Postgres restarted, creating the file.

Rerunning gitlab-rake gitlab:check I now get:

# gitlab-rake gitlab:check
Checking GitLab Shell ...

GitLab Shell version >= 2.6.5 ? ... OK (2.6.5)
Repo base directory exists? ... yes
Repo base directory is a symlink? ... no
Repo base owned by git:git? ... yes
Repo base access is drwxrws---? ... yes
hooks directories in repos are links: ... rake aborted!
PG::ConnectionBad: FATAL:  Peer authentication failed for user "gitlab"

And incase it matters:

# grep git /etc/passwd
git:x:999:999::/var/opt/gitlab:/bin/sh
gitlab-www:x:998:998::/var/opt/gitlab/nginx:/bin/false
gitlab-redis:x:997:997::/var/opt/gitlab/redis:/bin/nologin
gitlab-psql:x:996:996::/var/opt/gitlab/postgresql:/bin/sh

I’m running on Ubuntu 14.04.3, 64-bit
As far as I know I’m running the latest and greatest GitLab—though I don’t recall if I installed from source or via apt-get
Looks like I’m running PostgreSQL 9.3

I have a script to install security updates and reboot the server if required, it looks like the machine was rebooted yesterday morning. I last logged into the web interface about a week ago and everything was fine then.

Since everything has gone smoothly with GitLab, I’ve not really played with it, and although I have some DB experience it is not with Postgres…

Any troubleshooting help would be great.
Thanks,
Ben

2

Firstly, you are probably running the omnibus package. Check with dpkg -l gitlab-ce.

Now, GitLab uses its own bundled postgres so everything should work seamessly. If by any chance you are using an external posgres instance, check in /etc/gitlab/gitlab.rb if there are any relevant entries to these doc/settings/database.md · master · GitLab.org / omnibus-gitlab · GitLab

If not, stop the external database (if any) and rerun gitlab-ctl reconfigure.

What file exactly did you change?

1 Like
3

Thanks for your reply @axil, yes I do seem to have the omnibus package:

administrator@gitlab:~$ dpkg -l gitlab-ce
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                                          Version                     Architecture                Description
+++-=============================================-===========================-===========================-================================================================================================
ii  gitlab-ce                                     8.0.2-ce.1                  amd64                       GitLab Community Edition and GitLab CI (including NGINX, Postgres, Redis)

I haven’t overridden anything in /etc/gitlab/gitlab.rb so I’m guessing GitLab at least thinks it’s running its own Postgres instance.

I changed the unix_socket_directories setting in /etc/postgresql/9.3/main/postgresql.conf

I have now removed what appeared to be the Ubuntu supplied Postgres (apt-get remove postgresql) and rerun gitlab-ctl reconfigure, here’s a pastbin of the output.

I’m still getting the error, so reran gitlab-rake gitlab:check, that now fails on:

PG::ConnectionBad: could not connect to server: No such file or directory
	Is the server running locally and accepting
	connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?

Checking the status I can see that postgresql is down, but issuing a restart timeout when trying to bring it up:

root@gitlab:/tmp# gitlab-ctl status
run: gitlab-git-http-server: (pid 31563) 52s; run: log: (pid 1300) 265591s
run: logrotate: (pid 31571) 51s; run: log: (pid 1291) 265591s
run: nginx: (pid 31577) 51s; run: log: (pid 1281) 265591s
down: postgresql: 1s, normally up, want up; run: log: (pid 1282) 265591s
run: redis: (pid 31598) 44s; run: log: (pid 1287) 265591s
run: sidekiq: (pid 31603) 44s; run: log: (pid 1279) 265591s
run: unicorn: (pid 31623) 42s; run: log: (pid 1278) 265591s

root@gitlab:/tmp# gitlab-ctl restart
ok: run: gitlab-git-http-server: (pid 31818) 0s
ok: run: logrotate: (pid 31825) 1s
ok: run: nginx: (pid 31832) 0s
timeout: down: postgresql: 1s, normally up, want up
ok: run: redis: (pid 31901) 0s
ok: run: sidekiq: (pid 31906) 1s
ok: run: unicorn: (pid 31913) 0s

Looking in the logs for GitLabs Postgres, it showed that Postgres didn’t have the correct permissions for its directories—my fault! I had set it to the system Postgres. I gave gitlab-postgresql back its data and now it’s all back up and happy.

@axil thanks very much for your insight. It looks like Ubuntu Postgres got installed somehow and perhaps stopped the GitLab Postgres from starting, or something…Perhaps I’ll never know, but I’ll dig a bit more now that the service is back up.

Thanks once again,
Ben

4

Glad that you solved it :wink: