Gitlab suddenly not restarting - postgres locale problem?


I’ve been running gillab for a 3-4 months now. My Centos6 server needed a reboot and now I suddenly get the web page error 500 “whoops something went wrong”. When I run sudo gitlab-ctl reconfigure I get the following errors:

Expected process to exit with [0], but received ‘2’
---- Begin output of /opt/gitlab/bin/gitlab-psql -d template1 -c “CREATE USER gitlab” ----
STDERR: sh: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
/bin/sh: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
psql: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket “/var/opt/gitlab/postgresql/.s.PGSQL.5432”?
---- End output of /opt/gitlab/bin/gitlab-psql -d template1 -c “CREATE USER gitlab” ----
Ran /opt/gitlab/bin/gitlab-psql -d template1 -c “CREATE USER gitlab” returned 2

If I look for en_US.UTF-8 it’s there: locale -a | grep -i us shows it’s there, but there does seem to be a problem setting the locale to en_US.UTF-8 when I try elsewhere to use that locale. I’ve tried for days to fix this, but to no avail.

So questions:

  1. Is it correct to assume from the error message this is the main problem?
  2. Is there a way to work around the locale - i.e. rebuild the postgres database to the currently supported local?

Many thanks for guidance …


More info - gitlab does start and status reports everything is up, but gitlab-ctl tail reports huge repeat amount of error message:

2016-11-27_14:51:11.23005 DETAIL: The database was initialized with LC_COLLATE “en_US.UTF-8”, which is not recognized by setlocale().
2016-11-27_14:51:11.23006 HINT: Recreate the database with another locale or install the missing locale.
2016-11-27_14:51:11.24165 FATAL: database locale is incompatible with operating system

So that answers the first question.

Please let me know if I can rebuild the database to a different locale.


I just stepped on this land mine. Had something to do with another CentOS upgrade messing with the locales. I reran

localedef -v -c -i en_US -f UTF-8 en_US.UTF-8

and it magically started working.

Yes, this is the right way to re-generate a locale in case it got deleted.

FYI - my case was more complicated, someone changed /lib64/ to point to /opt/ so locale was actually looking in /opt/lib/locale or something like that and couldn’t find any installed locales. (warning - changing glibc can have very ugly side effects and is not recommended). Once I figured that out one should fix this libc link to point to back to original /lib64/

In this case I couldn’t change that back, so either put a link at /opt/xxx to /lib/usr/locale, or set LC_PATH to point to /lib/usr/locale/ . Both seem to work fine in my case.

Again, this is a lesson as to why one should not upgrade or mess with the default glibc libraries.