Update Omnibus to 7.11.2 fails because of missing DB connection (alternative port)

How to Use GitLab
1

Hi,

I just tried to update from 7.9.1 Omnibus to 7.11.2, but dpkg -i fails during DB migrations, saying that it cannot connect to the socket in /tmp. More explicitly:

Recipe: gitlab::database_migrations
  * bash[migrate gitlab-rails database] action run
    
    ================================================================================
    Error executing action `run` on resource 'bash[migrate gitlab-rails database]'
    ================================================================================
    
    Mixlib::ShellOut::ShellCommandFailed
    ------------------------------------
    Expected process to exit with [0], but received '1'
    ---- Begin output of "bash"  "/tmp/chef-script20150525-20857-c93yl2" ----
    STDOUT: rake aborted!
    PG::Error: could not connect to server: No such file or directory
    	Is the server running locally and accepting
    	connections on Unix domain socket "/tmp/.s.PGSQL.5433"?

Previously I already updated from 7.8 without any problem!

I hope this is not the problem, but I’m running Gitlab with different ports, e.g.

gitlab_rails['db_port'] = 5433

Reading problems of other people I also tried gitlab-rake db:migrate:status, and gitlab-rake gitlab:status. Basically everything fails with the same error.

I managed to remove gitlab 7.11.2 and install 7.9.1 and the system is again up and running, but I would like to be able to upgrade as usual.

Thanks.

2

Try to first start postgres. The database needs to be active in order for the migrations to run.

So first do a gitlab-ctl start postgresql, then try to update.

3

But the database is running already. Here are the commands I just tried.

Launch installation:

$ sudo dpkg -i gitlab-ce_7.11.2~omnibus-1_amd64.deb 
Selecting previously unselected package gitlab-ce.
dpkg: considering removing gitlab in favour of gitlab-ce ...
dpkg: yes, will remove gitlab in favour of gitlab-ce
(Reading database ... 278612 files and directories currently installed.)
Preparing to unpack gitlab-ce_7.11.2~omnibus-1_amd64.deb ...
Unpacking gitlab-ce (7.11.2~omnibus-1) ...
Setting up gitlab-ce (7.11.2~omnibus-1) ...
gitlab: Thank you for installing GitLab!
gitlab: Configure and start GitLab by running the following command:
gitlab: 
gitlab: sudo gitlab-ctl reconfigure
gitlab: 
gitlab: GitLab should be reachable at http://ipho2016-dev.physik.uzh.ch
gitlab: Otherwise configure GitLab for your system by editing /etc/gitlab/gitlab.rb file
gitlab: And running reconfigure again.
gitlab: 
gitlab: For a comprehensive list of configuration options please see the Omnibus GitLab readme
gitlab: https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/README.md
gitlab: 
Found /etc/gitlab/skip-auto-migrations, exiting...

Check running databases:

$ ps aux | grep postgresql
postgres  9914  0.0  1.0 247740 21196 ?        S    06:45   0:00 /usr/lib/postgresql/9.3/bin/postgres -D /var/lib/postgresql/9.3/main -c config_file=/etc/postgresql/9.3/main/postgresql.conf
gitlab-+ 27233  0.0  1.9 576280 39412 ?        Ss   May25   0:00 /opt/gitlab/embedded/bin/postgres -D /var/opt/gitlab/postgresql/data

Start DB as you sugegsted:

$ sudo gitlab-ctl start postgresql
ok: run: postgresql: (pid 27233) 60370s

Run the reconfigure:

$ sudo gitlab-ctl reconfigure
Starting Chef Client, version 11.18.0
resolving cookbooks for run list: ["gitlab"]
Synchronizing Cookbooks:
  - gitlab
  - runit
Compiling Cookbooks...
...
...
Recipe: gitlab::database_migrations
  * bash[migrate gitlab-rails database] action run
    
    ================================================================================
    Error executing action `run` on resource 'bash[migrate gitlab-rails database]'
    ================================================================================
    
    Mixlib::ShellOut::ShellCommandFailed
    ------------------------------------
    Expected process to exit with [0], but received '1'
    ---- Begin output of "bash"  "/tmp/chef-script20150526-1107-ywqx83" ----
    STDOUT: rake aborted!
    PG::Error: could not connect to server: No such file or directory
    	Is the server running locally and accepting
    	connections on Unix domain socket "/tmp/.s.PGSQL.5433"?

As you see, the process id of the postgresql instance running is the same as the one reported by gitlab-ctl start postgresql.

It looks to me that it tries to connect to the wrong socket, because in the running 7.9.1 installation there is never such a file in /tmp, they must be somewhere else.

4

I also tried to restart postgresql.
This time the reconfigure runs until the end, but I think it is doing nothing:

Recipe: gitlab::database_migrations
  * execute[initialize gitlab-rails database] action nothing (skipped due to action :nothing)
  * execute[initialize gitlab-ci database] action nothing (skipped due to action :nothing)
  * bash[migrate gitlab-rails database] action nothing (skipped due to action :nothing)
  * bash[migrate gitlab-ci database] action nothing (skipped due to action :nothing)

Running a status check fails as before.

5

I managed to reproduce the problem also with a clean installation, where I specify a custom value for the configuration gitlab_rails['db_port'].

Extending my search to installation problems and not only upgrade, I found out that my gitlab.rb configuration might be broken, but somehow earlier versions of Gitlab didn’t care about that bug.
Now I think that the problem is that one should also modify the postgresql['port'] configuration.

I managed to install a fresh copy of Gitlab 7.11.4 with the following settings:

postgresql['enable'] = true
postgresql['port'] = 5433
gitlab_rails['db_port'] = 5433

For the upgrade to work correctly it was then sufficient to add the missing configuration before upgrading.

6

Interesting, we should document this. Thanks for the investigation.

7

If it helps, I don’t find the post anymore, but I was reading that a bug has (recently) been fixed about the postresql port not being set correctly.