Errors in logs after update: There is an issue connecting to your database with your username/password, username

Problem to solve

After update Gitlab-docker from version 16.10.3 to version 16.11.4, we receives the following logs moment after the service starts:

There is an issue connecting to your database with your username/password, username: gitlab
{"severity":"ERROR","time":"2024-06-13T10:33:10.509Z","correlation_id":"01J0BSH1TQXWMBC9F8Y4F15A7S","exception.class":"ActiveRecord::DatabaseConnectionError","exception.message":"There is an issue connecting to your database with your username/password, username: gitlab.\n\nPlease check your database configuration to ensure the username/password are valid.\n","exception.backtrace":["activerecord (7.0.8.1) lib/active_record/connection_adapters/postgresql_adapter.rb:83:in `rescue in new_client'","activerecord (7.0.8.1) lib/active_record/connection_adapters/postgresql_adapter.rb:77:in `new_client'","activerecord (7.0.8.1) lib/active_record/connection_adapters/postgresql_adapter.rb:37:in `postgresql_connection'","activerecord (7.0.8.1) lib/active_record/connection_adapters/abstract/connection_pool.rb:656:in `public_send'","activerecord (7.0.8.1) lib/active_record/connection_adapters/abstract/connection_pool.rb:656:in `new_connection'","activerecord (7.0.8.1) lib/active_record/connection_adapters/abstract/connection_pool.rb:700:in `checkout_new_connection'","activerecord (7.0.8.1) lib/active_record/connection_adapters/abstract/connection_pool.rb:679:in `try_to_checkout_new_connection'","activerecord (7.0.8.1) lib/active_record/connection_adapters/abstract/connection_pool.rb:640:in `acquire_connection'","activerecord (7.0.8.1) lib/active_record/connection_adapters/abstract/connection_pool.rb:341:in `checkout'","activerecord (7.0.8.1) lib/active_record/connection_adapters/abstract/connection_pool.rb:181:in `connection'","lib/gitlab/database/load_balancing/primary_host.rb:44:in `connection'","lib/gitlab/database/load_balancing/load_balancer.rb:64:in `read'","lib/gitlab/database/load_balancing/connection_proxy.rb:67:in `schema_cache'","activerecord (7.0.8.1) lib/active_record/model_schema.rb:407:in `table_exists?'","lib/gitlab/database/query_analyzers/log_large_in_lists.rb:24:in `enabled?'","lib/gitlab/database/query_analyzer.rb:52:in `block in begin!'","lib/gitlab/database/query_analyzer.rb:49:in `select'","lib/gitlab/database/query_analyzer.rb:49:in `begin!'","lib/gitlab/database/query_analyzer.rb:37:in `within'","lib/gitlab/middleware/query_analyzer.rb:11:in `call'","lib/gitlab/middleware/organizations/current.rb:20:in `call'","batch-loader (2.0.1) lib/batch_loader/middleware.rb:11:in `call'","rack-attack (6.7.0) lib/rack/attack.rb:103:in `call'","apollo_upload_server (2.1.5) lib/apollo_upload_server/middleware.rb:19:in `call'","lib/gitlab/middleware/multipart.rb:173:in `call'","rack-attack (6.7.0) lib/rack/attack.rb:127:in `call'","warden (1.2.9) lib/warden/manager.rb:36:in `block in call'","warden (1.2.9) lib/warden/manager.rb:34:in `catch'","warden (1.2.9) lib/warden/manager.rb:34:in `call'","rack-cors (2.0.1) lib/rack/cors.rb:102:in `call'","rack (2.2.8.1) lib/rack/tempfile_reaper.rb:15:in `call'","rack (2.2.8.1) lib/rack/etag.rb:27:in `call'","rack (2.2.8.1) lib/rack/conditional_get.rb:40:in `call'","rack (2.2.8.1) lib/rack/head.rb:12:in `call'","actionpack (7.0.8.1) lib/action_dispatch/http/permissions_policy.rb:38:in `call'","actionpack (7.0.8.1) lib/action_dispatch/http/content_security_policy.rb:36:in `call'","lib/gitlab/middleware/read_only/controller.rb:50:in `call'","lib/gitlab/middleware/read_only.rb:18:in `call'","lib/gitlab/middleware/unauthenticated_session_expiry.rb:18:in `call'","rack (2.2.8.1) lib/rack/session/abstract/id.rb:266:in `context'","rack (2.2.8.1) lib/rack/session/abstract/id.rb:260:in `call'","actionpack (7.0.8.1) lib/action_dispatch/middleware/cookies.rb:704:in `call'","lib/gitlab/middleware/same_site_cookies.rb:27:in `call'","actionpack (7.0.8.1) lib/action_dispatch/middleware/callbacks.rb:27:in `block in call'","activesupport (7.0.8.1) lib/active_support/callbacks.rb:99:in `run_callbacks'","actionpack (7.0.8.1) lib/action_dispatch/middleware/callbacks.rb:26:in `call'","sentry-rails (5.10.0) lib/sentry/rails/rescued_exception_interceptor.rb:12:in `call'","actionpack (7.0.8.1) lib/action_dispatch/middleware/debug_exceptions.rb:28:in `call'","lib/gitlab/middleware/path_traversal_check.rb:35:in `call'","lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'","sentry-ruby (5.10.0) lib/sentry/rack/capture_exceptions.rb:28:in `block (2 levels) in call'","sentry-ruby (5.10.0) lib/sentry/hub.rb:227:in `with_session_tracking'","sentry-ruby (5.10.0) lib/sentry-ruby.rb:385:in `with_session_tracking'","sentry-ruby (5.10.0) lib/sentry/rack/capture_exceptions.rb:19:in `block in call'","sentry-ruby (5.10.0) lib/sentry/hub.rb:59:in `with_scope'","sentry-ruby (5.10.0) lib/sentry-ruby.rb:365:in `with_scope'","sentry-ruby (5.10.0) lib/sentry/rack/capture_exceptions.rb:18:in `call'","actionpack (7.0.8.1) lib/action_dispatch/middleware/show_exceptions.rb:29:in `call'","lib/gitlab/middleware/basic_health_check.rb:25:in `call'","lograge (0.11.2) lib/lograge/rails_ext/rack/logger.rb:15:in `call_app'","railties (7.0.8.1) lib/rails/rack/logger.rb:25:in `block in call'","activesupport (7.0.8.1) lib/active_support/tagged_logging.rb:99:in `block in tagged'","activesupport (7.0.8.1) lib/active_support/tagged_logging.rb:37:in `tagged'","activesupport (7.0.8.1) lib/active_support/tagged_logging.rb:99:in `tagged'","railties (7.0.8.1) lib/rails/rack/logger.rb:25:in `call'","actionpack (7.0.8.1) lib/action_dispatch/middleware/remote_ip.rb:93:in `call'","lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'","lib/gitlab/middleware/request_context.rb:15:in `call'","lib/gitlab/middleware/webhook_recursion_detection.rb:15:in `call'","request_store (1.5.1) lib/request_store/middleware.rb:19:in `call'","rack (2.2.8.1) lib/rack/method_override.rb:24:in `call'","rack (2.2.8.1) lib/rack/runtime.rb:22:in `call'","rack-timeout (0.6.3) lib/rack/timeout/core.rb:148:in `block in call'","rack-timeout (0.6.3) lib/rack/timeout/support/timeout.rb:19:in `timeout'","rack-timeout (0.6.3) lib/rack/timeout/core.rb:147:in `call'","config/initializers/fix_local_cache_middleware.rb:11:in `call'","lib/gitlab/middleware/compressed_json.rb:44:in `call'","actionpack (7.0.8.1) lib/action_dispatch/middleware/executor.rb:14:in `call'","lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call'","rack (2.2.8.1) lib/rack/sendfile.rb:110:in `call'","lib/gitlab/middleware/sidekiq_web_static.rb:20:in `call'","actionpack (7.0.8.1) lib/action_dispatch/middleware/host_authorization.rb:131:in `call'","lib/gitlab/metrics/requests_rack_middleware.rb:79:in `call'","gitlab-labkit (0.35.1) lib/labkit/middleware/rack.rb:22:in `block in call'","gitlab-labkit (0.35.1) lib/labkit/context.rb:35:in `with_context'","gitlab-labkit (0.35.1) lib/labkit/middleware/rack.rb:21:in `call'","actionpack (7.0.8.1) lib/action_dispatch/middleware/request_id.rb:26:in `call'","sentry-raven (3.1.2) lib/raven/integrations/rack.rb:51:in `call'","railties (7.0.8.1) lib/rails/engine.rb:530:in `call'","railties (7.0.8.1) lib/rails/railtie.rb:226:in `public_send'","railties (7.0.8.1) lib/rails/railtie.rb:226:in `method_missing'","lib/gitlab/middleware/release_env.rb:13:in `call'","rack (2.2.8.1) lib/rack/urlmap.rb:74:in `block in call'","rack (2.2.8.1) lib/rack/urlmap.rb:58:in `each'","rack (2.2.8.1) lib/rack/urlmap.rb:58:in `call'","puma (6.4.0) lib/puma/configuration.rb:272:in `call'","puma (6.4.0) lib/puma/request.rb:100:in `block in handle_request'","puma (6.4.0) lib/puma/thread_pool.rb:378:in `with_force_shutdown'","puma (6.4.0) lib/puma/request.rb:99:in `handle_request'","puma (6.4.0) lib/puma/server.rb:443:in `process_client'","puma (6.4.0) lib/puma/server.rb:241:in `block in run'","puma (6.4.0) lib/puma/thread_pool.rb:155:in `block in spawn_thread'"],"exception.cause_class":"PG::ConnectionBad","user.username":null,"tags.program":"web","tags.locale":"en","tags.feature_category":null,"tags.correlation_id":"01J0BSH1TQXWMBC9F8Y4F15A7S"}

What’s the reason?
After boot, there are no more such errors and it looks like everything works fine
I checked that all background_migrations were successful

Steps to reproduce

Update from 16.10.3 to 16.11.4

Configuration

I use the default Postgres configuration built into the Gitlab docker (except for the prometheus metrics endpoint)

Versions

16.10.3 > 16.11.4