Postgreql server is down and gitlab url is not accesible

Hi Team,

We are using GITLAB application which is installed in Linux machine. We are accessing it through web portal (http://machinename:port). GITLAB went down with error code(“500” ) in browser, after We have changed the port number in the external url defined in the file “gitlab.rb” (/etc/gitlab) . the status of dbserver “postgresql” shows down while restart the GITLAB application but the same time postgresql process is running from “root” user and we are not able to delete them as it is getting created again and again. Please suggest what needs to be done to bring it up “postgresql” server and gitlab portal.

Status of postgresql :

But postgresql process is running :

We tried the below fixes but it doesn’t help:

  1. Tried to change “shared_buffers” size
  2. Reboot machine several times with existing and changed port numbers
  3. Ran Reconfigure and restart commands
  4. Deleted the postmaster.pid (/var/opt/gitlab/postgresql/data ) and restarted the application.

Permission we have on folders:

/var/opt/gitlab/postgresql

Error we are getting:

2016-04-13_15:37:27.21627 FATAL: could not open lock file “/var/opt/gitlab/postgresql/.s.PGSQL.5432.lock”: Permission denied
2016-04-13_15:37:28.22597 FATAL: could not open lock file “/var/opt/gitlab/postgresql/.s.PGSQL.5432.lock”: Permission denied
2016-04-13_15:37:29.23636 FATAL: could not open lock file “/var/opt/gitlab/postgresql/.s.PGSQL.5432.lock”: Permission denied
2016-04-13_15:37:30.24846 FATAL: could not open lock file “/var/opt/gitlab/postgresql/.s.PGSQL.5432.lock”: Permission denied
2016-04-13_15:37:31.25965 FATAL: could not open lock file “/var/opt/gitlab/postgresql/.s.PGSQL.5432.lock”: Permission denied

Production log Error trace:

Started GET “/users/sign_in” for 127.0.0.1 at 2016-04-13 16:00:28 +0530

PG::ConnectionBad (could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket “/var/opt/gitlab/postgresql/.s.PGSQL.5432”?
):
activerecord (4.1.12) lib/active_record/connection_adapters/postgresql_adapter.rb:888:in initialize' activerecord (4.1.12) lib/active_record/connection_adapters/postgresql_adapter.rb:888:in new’
activerecord (4.1.12) lib/active_record/connection_adapters/postgresql_adapter.rb:888:in connect' activerecord (4.1.12) lib/active_record/connection_adapters/postgresql_adapter.rb:568:in initialize’
activerecord (4.1.12) lib/active_record/connection_adapters/postgresql_adapter.rb:41:in new' activerecord (4.1.12) lib/active_record/connection_adapters/postgresql_adapter.rb:41:in postgresql_connection’
activerecord (4.1.12) lib/active_record/connection_adapters/abstract/connection_pool.rb:435:in new_connection' activerecord (4.1.12) lib/active_record/connection_adapters/abstract/connection_pool.rb:445:in checkout_new_connection’
activerecord (4.1.12) lib/active_record/connection_adapters/abstract/connection_pool.rb:416:in acquire_connection' activerecord (4.1.12) lib/active_record/connection_adapters/abstract/connection_pool.rb:351:in block in checkout’
/opt/gitlab/embedded/lib/ruby/2.1.0/monitor.rb:211:in `mon_synchronize’

Current log error trace:

2016-04-13_11:35:39.24275 FATAL: pre-existing shared memory block (key 5432001, ID 0) is still in use
2016-04-13_11:35:39.24277 HINT: If you’re sure there are no old server processes still running, remove the shared memory block or just delete the file “postmaster.pid”.
2016-04-13_11:35:40.25068 FATAL: pre-existing shared memory block (key 5432001, ID 0) is still in use
2016-04-13_11:35:40.25070 HINT: If you’re sure there are no old server processes still running, remove the shared memory block or just delete the file “postmaster.pid”.
2016-04-13_11:35:41.25899 FATAL: pre-existing shared memory block (key 5432001, ID 0) is still in use
2016-04-13_11:35:41.25901 HINT: If you’re sure there are no old server processes still running, remove the shared memory block or just delete the file “postmaster.pid”.
2016-04-13_11:35:42.26682 FATAL: pre-existing shared memory block (key 5432001, ID 0) is still in use
2016-04-13_11:35:42.26684 HINT: If you’re sure there are no old server processes still running, remove the shared memory block or just delete the file “postmaster.pid”.

Regards
Suresh G

@sureshG, How did you fix this? I facing the same issue.

anyone has any updates on this ? @arunkumar @sureshG