Bad connection to postgres during update and some more errors after migrating to 9.2.x-ce.0 / docker image on centos-atomic

We’ve been running a gitlab 8.16.3-ce.1 for a while and want to upgrade. Following the steps in the docs [1], I just pulled the new docker image (first 9.2.4 and later 9.2.5 when it came out) and ran it (on centos-atomic, with the docker run command that includes de ‘:Z’ in de volume mounts, for selinux)

regardless of whether there is prior data, The first few minutes, I get this error a lot:

PG::ConnectionBad (could not connect to server: No such file or directory
    Is the server running locally and accepting
    connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?
):

After waiting a bit, gitlab seems to run ok. The gui is up and responds to commands. I can see the projects I expect.

However, the logging contains these errors:

  • 1: /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/current_settings.rb:28:in `sidekiq_throttling_enabled?’
    (happens thousends of times every so many seconds)
  • 2: ERROR: column projects.mirror does not exist at character 57
    (followed by a select statement. happens every 15 seconds)
  • 3: ERROR: wifi collector failed after 0.007549s: failed to access wifi data: permission denied" source=“node_exporter.go:95”
    (every 15 seconds)

I found a couple of resources that helped.

  • The sidekiq-message (1) disappears after running gitlab-ctl reconfigure.
  • the ‘column does not exist’-message (2) seems to be related to a bug in gitlab monitor, that needs to be fixed [2]
  • following a hunch, I disabled selinux on the host os and the wifi error (3) does not appear anymore
  • according to [3] and [4], the issue with the bad database connection can be fixed by setting the /opt/gitlab/embedded/service/gitlab-rails/config/database.yml ‘host:’ property to be empty, instead of /var/opt/gitlab/postgresql (currently, there’s a couple of .lock files there)

We roll out a gitlab instance quite often, so what I hope to get from you, is:

  • what effect could the ‘bad connection’ have in the end? What can I do to get around it?
  • I’m using docker images to minimize configuration issues and don’t want to run gitlab-ctl reconfigure after rolling out the image. I don’t think that’s the normal flow. What causes the sidekiq errors in the first place?
  • I’d like the configure selinux to be able to work without errors. What permissions do I need to set to enable the wifi data collection that fails now?

[1] https://docs.gitlab.com/omnibus/docker/README.html
[2] https://gitlab.com/gitlab-org/gitlab-monitor/issues/25
[3] https://gitlab.com/gitlab-org/gitlab-ce/issues/2005
[4] https://stackoverflow.com/questions/40005023/gitlab-500-error-pgconnectionbad

update:

the wifi error message (caused by selinux) can be fixed by setting:
setsebool -P virt_sandbox_use_netlink 1

in docker compose, that would be:

  • name: Enable selinux boolean for using netlink, to prevent an wifi data error
    seboolean:
    name: virt_sandbox_use_netlink
    state: yes
    persistent: yes
    become: true

update2: this also fixes the PG: Bad connection error and the sidekiq error (1). The only error left is the one with the missing column, which seems to require a fix from gitlab itself.

Hope this helps someone else :slight_smile:

there’s one tiny thing left: lots and lots of occurences of this message:
==> /var/log/gitlab/gitlab-workhorse/current <==
2017-06-13_15:04:52.30330 gitlab-1.shared.dev.quattro @ - - [2017-06-13 15:04:52.297942243 +0000 UTC] “POST /ci/api/v1/builds/register.json HTTP/1.1” 204 0 “” “gitlab-ci-multi-runner 1.10.3 (1-10-stable; go1.7.4; linux/amd64)” 0.005235

I don’t want to clog the logging with ok messages from the runner if it can be avoided. Doesn’t seem to be effected by the general log_level setting in production.rb