GitLab Mattermost connection refused after update to 14.9.1

How to Use GitLab Self-managed
,
1

Hello,

We use GitLab EE and GitLab’s embedded mattermost as two different services configured in a docker-compose.yml file. Until now, there was no update issue, following tag-by-tag the GitLab releases.
Today, by upgrading from 14.8.4 to 14.9.1, the Mattermost service fails to start because of a connection error:

==> /var/log/gitlab/nginx/gitlab_mattermost_error.log <==
2022/03/29 17:17:08 [error] 124#0: *7 connect() failed (111: Connection refused) while connecting to upstream, client: 172.18.0.1, server: mattermost.myhost.com, request: "GET /api/v4/plugins/webapp HTTP/1.1", upstream: "http://127.0.0.1:8065/api/v4/plugins/webapp", host: "mattermost.myhost.com"

==> /var/log/gitlab/nginx/gitlab_mattermost_access.log <==
172.18.0.1 - - [29/Mar/2022:17:17:08 +0000] "GET /api/v4/plugins/webapp HTTP/1.1" 502 150 "" "Mozilla/5.0 (X11; Linux x86_64; rv:98.0) Gecko/20100101 Firefox/98.0" -

==> /var/log/gitlab/nginx/gitlab_mattermost_error.log <==
2022/03/29 17:17:08 [error] 124#0: *7 connect() failed (111: Connection refused) while connecting to upstream, client: 172.18.0.1, server: mattermost.myhost.com, request: "GET /api/v4/users/me/teams/cfxtt5apgigkics3q3qde99esr/channels?include_deleted=true HTTP/1.1", upstream: "http://127.0.0.1:8065/api/v4/users/me/teams/cfxtt5apgigkics3q3qde99esr/channels?include_deleted=true", host: "mattermost.myhost.com"

==> /var/log/gitlab/nginx/gitlab_mattermost_access.log <==
172.18.0.1 - - [29/Mar/2022:17:17:08 +0000] "GET /api/v4/users/me/teams/cfxtt5apgigkics3q3qde99esr/channels?include_deleted=true HTTP/1.1" 502 150 "" "Mozilla/5.0 (X11; Linux x86_64; rv:98.0) Gecko/20100101 Firefox/98.0" -

==> /var/log/gitlab/nginx/gitlab_mattermost_error.log <==
2022/03/29 17:17:08 [error] 124#0: *7 connect() failed (111: Connection refused) while connecting to upstream, client: 172.18.0.1, server: mattermost.myhost.com, request: "GET /api/v4/users/me/teams/cfxtt5apgigkics3q3qde99esr/channels/members HTTP/1.1", upstream: "http://127.0.0.1:8065/api/v4/users/me/teams/cfxtt5apgigkics3q3qde99esr/channels/members", host: "mattermost.myhost.com"
2022/03/29 17:17:08 [error] 125#0: *8 connect() failed (111: Connection refused) while connecting to upstream, client: 172.18.0.1, server: mattermost.myhost.com, request: "GET /api/v4/channels/inh466jcwp8ezqzonoj98n768w/posts?since=1648574094050&skipFetchThreads=false&collapsedThreads=false&collapsedThreadsExtended=false HTTP/1.1", upstream: "http://127.0.0.1:8065/api/v4/channels/inh466jcwp8ezqzonoj98n768w/posts?since=1648574094050&skipFetchThreads=false&collapsedThreads=false&collapsedThreadsExtended=false", host: "mattermost.myhost.com"

==> /var/log/gitlab/nginx/gitlab_mattermost_access.log <==
172.18.0.1 - - [29/Mar/2022:17:17:08 +0000] "GET /api/v4/users/me/teams/cfxtt5apgigkics3q3qde99esr/channels/members HTTP/1.1" 502 150 "" "Mozilla/5.0 (X11; Linux x86_64; rv:98.0) Gecko/20100101 Firefox/98.0" -
172.18.0.1 - - [29/Mar/2022:17:17:08 +0000] "GET /api/v4/channels/inh466jcwp8ezqzonoj98n768w/posts?since=1648574094050&skipFetchThreads=false&collapsedThreads=false&collapsedThreadsExtended=false HTTP/1.1" 502 150 "" "Mozilla/5.0 (X11; Linux x86_64; rv:98.0) Gecko/20100101 Firefox/98.0" -

==> /var/log/gitlab/nginx/gitlab_mattermost_error.log <==
2022/03/29 17:17:08 [error] 125#0: *8 connect() failed (111: Connection refused) while connecting to upstream, client: 172.18.0.1, server: mattermost.myhost.com, request: "GET /api/v4/users/me/teams/unread?include_collapsed_threads=false HTTP/1.1", upstream: "http://127.0.0.1:8065/api/v4/users/me/teams/unread?include_collapsed_threads=false", host: "mattermost.myhost.com"
2022/03/29 17:17:08 [error] 124#0: *7 connect() failed (111: Connection refused) while connecting to upstream, client: 172.18.0.1, server: mattermost.myhost.com, request: "POST /api/v4/users/ids?since=1648574148655 HTTP/1.1", upstream: "http://127.0.0.1:8065/api/v4/users/ids?since=1648574148655", host: "mattermost.myhost.com"

==> /var/log/gitlab/nginx/gitlab_mattermost_access.log <==
172.18.0.1 - - [29/Mar/2022:17:17:08 +0000] "GET /api/v4/users/me/teams/unread?include_collapsed_threads=false HTTP/1.1" 502 150 "" "Mozilla/5.0 (X11; Linux x86_64; rv:98.0) Gecko/20100101 Firefox/98.0" -
172.18.0.1 - - [29/Mar/2022:17:17:08 +0000] "POST /api/v4/users/ids?since=1648574148655 HTTP/1.1" 502 150 "" "Mozilla/5.0 (X11; Linux x86_64; rv:98.0) Gecko/20100101 Firefox/98.0" -

==> /var/log/gitlab/alertmanager/current <==
2022-03-29_17:17:09.15155 level=info ts=2022-03-29T17:17:09.151Z caller=cluster.go:688 component=cluster msg="gossip settled; proceeding" elapsed=10.002716828s

This pattern repeats for a bunch of URLs and the service stay in unhealthy state.

The configuration didn’t change and we mostly use the standard parameters. Here is a part from our docker-compose.yml file:

version: '2'

services:
  postgresql:
    ...

  gitlab:
    ...

  mattermost:
    restart: on-failure
    image: gitlab/gitlab-ee:14.9.1-ee.0
    depends_on:
    - postgresql
    - gitlab
    ports:
    - "80:80"
    volumes:
      - '/srv/gitlab/mattermost/config:/etc/gitlab'
      - '/srv/gitlab/mattermost/logs:/var/log/'
      - '/srv/gitlab/mattermost/data:/var/opt/gitlab'
    environment:
      GITLAB_OMNIBUS_CONFIG: |
        mattermost_external_url 'https://mattermost.myhost.com'
        postgresql['enable'] = false
        gitlab_rails['enable'] = false
        mattermost['sql_driver_name'] = 'postgres'
        mattermost['sql_data_source'] = "<url>"
        mattermost_nginx['listen_port'] = 80
        mattermost_nginx['listen_https'] = false
        mattermost_nginx['proxy_set_headers'] = {
          "X-Forwarded-Proto" => "https",
          "X-Forwarded-Ssl" => "on"
        }
        mattermost['team_site_name'] = "Mattermost"
        mattermost['gitlab_enable'] = true
        mattermost['gitlab_id'] = "<id>"
        mattermost['gitlab_secret'] = "<secret>"
        mattermost['gitlab_scope'] = ""
        mattermost['gitlab_auth_endpoint'] = "https://gitlab.myhost.com/oauth/authorize"
        mattermost['gitlab_token_endpoint'] = "https://gitlab.myhost.com/oauth/token"
        mattermost['gitlab_user_api_endpoint'] = "https://gitlab.myhost.com/api/v4/user"
        mattermost['email_enable_sign_in_with_email'] = false
        mattermost['email_enable_sign_up_with_email'] = false

Note that the gitlab service successfully started on 14.9.1. The mattermost service also fails with 14.9.0, but works fine in 14.8.4.

We don’t use custom configuration related to let’s encrypt, and don’t have any config uncommented in gitlab.rb.

It seems to be related to the internal nginx server of the GitLab’s embedded Mattermost but at this point I’m unable to figure out what is causing this problem.

Thanks for any help.