GL CE external postgresql TLS configuration not being applied

Problem to solve

Docker version of Omnibus GitLab-CE is attempting to connect to an external Postgresql server without TLS certificates.

  • Postgresql 16.3-alpine on separate host

Steps to reproduce

Database was properly initialized, so TLS configuration was honored for that process. The ouput of gitlab-rails dbconsole --database main show TLS as configured:

SSL connection (protocol: TLSv1.3, cipher: TLS_AES_256_GCM_SHA384, bits: 256, compression: off)
Type "help" for help.

gitlabhq_production=# SELECT * FROM pg_stat_ssl;
  pid   | ssl | version |         cipher         | bits |                            client_dn
  |                 client_serial                 |                   issuer_dn
--------+-----+---------+------------------------+------+---------------------------------------------------------------
--+-----------------------------------------------+------------------------------------------------
 243329 | t   | TLSv1.3 | TLS_AES_256_GCM_SHA384 |  256 | REDACTED | REDACTED  | REDACTED
 241987 | t   | TLSv1.3 | TLS_AES_256_GCM_SHA384 |  256 | REDACTED | REDACTED  | REDACTED
...
 243105 | t   | TLSv1.3 | TLS_AES_256_GCM_SHA384 |  256 | REDACTED | REDACTED | REDACTED
(24 rows)

The console logs show connection attempts to the external Postgresql server with out TLS

gitlab  | ==> /var/log/gitlab/gitlab-exporter/current <==
gitlab  | 2024-07-31_15:52:36.04084 ::1 - - [31/Jul/2024:15:52:36 UTC] "GET /sidekiq HTTP/1.1" 200 509
gitlab  | 2024-07-31_15:52:36.04103 - -> /sidekiq
gitlab  | 2024-07-31_15:52:41.47630 E, [2024-07-31T15:52:41.475638 #963] ERROR -- : Bad connection to the database, resetting pool: connection to server at "10.0.0.151", port 15432 failed: FATAL:  connection requires a valid client certificate
gitlab  | 2024-07-31_15:52:41.47642 connection to server at "10.0.0.151", port 15432 failed: FATAL:  no pg_hba.conf entry for host "10.0.0.110", user "gitlab", database "gitlabhq_production", no encryption
gitlab  | 2024-07-31_15:52:41.47708
gitlab  | 2024-07-31_15:52:41.49127 E, [2024-07-31T15:52:41.490512 #963] ERROR -- : Bad connection to the database, resetting pool: connection to server at "10.0.0.151", port 15432 failed: FATAL:  connection requires a valid client certificate
gitlab  | 2024-07-31_15:52:41.49131 connection to server at "10.0.0.151", port 15432 failed: FATAL:  no pg_hba.conf entry for host "10.0.0.110", user "gitlab", database "gitlabhq_production", no encryption
gitlab  | 2024-07-31_15:52:41.49132
gitlab  | 2024-07-31_15:52:41.50729 E, [2024-07-31T15:52:41.506496 #963] ERROR -- : Bad connection to the database, resetting pool: connection to server at "10.0.0.151", port 15432 failed: FATAL:  connection requires a valid client certificate
gitlab  | 2024-07-31_15:52:41.50734 connection to server at "10.0.0.151", port 15432 failed: FATAL:  no pg_hba.conf entry for host "10.0.0.110", user "gitlab", database "gitlabhq_production", no encryption
gitlab  | 2024-07-31_15:52:41.50735
gitlab  | 2024-07-31_15:52:41.52653 ::1 - - [31/Jul/2024:15:52:41 UTC] "GET /database HTTP/1.1" 200 0
gitlab  | 2024-07-31_15:52:41.52659 - -> /database

Configuration

gitlab.rb relevant section

 postgresql['enable'] = false
 gitlab_rails['db_adapter'] = 'postgresql'
 gitlab_rails['db_encoding'] = 'utf8'
 gitlab_rails['db_host'] = 'postgresql05'
 gitlab_rails['db_port'] = 15432
 gitlab_rails['db_username'] = 'gitlab'
 gitlab_rails['db_password'] = "REDACTED"
 gitlab_rails['db_sslmode'] = 'require'
 gitlab_rails['db_sslkey'] = '/data/tls/gitlab.key'
 gitlab_rails['db_sslcert'] = '/data/tls/gitlab.crt'
 gitlab_rails['db_sslrootcert'] = '/data/tls/ca.cer'

pg_hba.conf (for reference)

# TYPE  DATABASE        USER            ADDRESS                 AUTH-METHOD  AUTH-OPTIONS

# "local" is for Unix domain socket connections only
local   all             all                                     trust

# IPv4 local connections:
host    all             all             127.0.0.1/32            trust

# IPv6 local connections:
host    all             all             ::1/128                 trust

# Allow replication connections from localhost, by a user with the
# replication privilege.
local   replication     all                                     trust
host    replication     all             127.0.0.1/32            trust
host    replication     all             ::1/128                 trust

# SSL
hostssl all             all             0.0.0.0/0               cert         #clientcert=verify-ca

Versions

Please select whether options apply, and add the version information.

Versions

  • Docker image gitlab-ce:172.2.0-ce.0

Thanks for taking the time to review this issue :blush:

Update 08-08-2024:
Docker image used is now gitlab/gitlab-ce:17.2.2-ce.0
Tried to change the gitlab.rb settings for gitlab-rails to

 gitlab_rails['db_sslmode'] = "verify-full"
 gitlab_rails['db_sslkey'] = "/data/tls/gitlab.key"
 gitlab_rails['db_sslcert'] = "/data/tls/gitlab.crt"
 gitlab_rails['db_sslrootcert'] = "/data/tls/ca.cer"

And changed the values to be enclosed in double quotes instead of single quotes.

The result is still no proper connection to the external database

2024-08-08_18:53:41.47287 E, [2024-08-08T18:53:41.471470 #344] ERROR -- : Bad connection to the database, resetting pool: connection to server at "10.0.0.151", port 15432 failed: FATAL:  connection requires a valid client certificate
2024-08-08_18:53:41.47291 connection to server at "10.0.0.151", port 15432 failed: FATAL:  no pg_hba.conf entry for host "192.168.230.110", user "gitlab", database "gitlabhq_production", no encryption
2024-08-08_18:53:41.47292
2024-08-08_18:53:41.48881 E, [2024-08-08T18:53:41.487497 #344] ERROR -- : Bad connection to the database, resetting pool: connection to server at "10.0.0.151", port 15432 failed: FATAL:  connection requires a valid client certificate
2024-08-08_18:53:41.48910 connection to server at "10.0.0.151", port 15432 failed: FATAL:  no pg_hba.conf entry for host "192.168.230.110", user "gitlab", database "gitlabhq_production", no encryption
2024-08-08_18:53:41.49002
2024-08-08_18:53:41.50425 E, [2024-08-08T18:53:41.501359 #344] ERROR -- : Bad connection to the database, resetting pool: connection to server at "10.0.0.151", port 15432 failed: FATAL:  connection requires a valid client certificate
2024-08-08_18:53:41.50427 connection to server at "10.0.0.151", port 15432 failed: FATAL:  no pg_hba.conf entry for host "192.168.230.110", user "gitlab", database "gitlabhq_production", no encryption