We upgraded gitlab CE to 16.4.2 about 3 days ago, it was working fine until this morning, when some of my colleagues were getting 500 sometimes.
After searching production.log we found
lib/gitlab/database/load_balancing/load_balancer.rb:120:in `read_write'
lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `write_using_load_balancer'
lib/gitlab/database/load_balancing/connection_proxy.rb:96:in `method_missing'
app/finders/ci/pipelines_finder.rb:173:in `sort_items'
app/finders/ci/pipelines_finder.rb:43:in `execute'
app/graphql/resolvers/project_pipeline_resolver.rb:34:in `block in resolve'
lib/gitlab/graphql/tracers/timer_tracer.rb:20:in `trace'
lib/gitlab/graphql/tracers/logger_tracer.rb:14:in `trace'
lib/gitlab/graphql/tracers/metrics_tracer.rb:13:in `trace'
lib/gitlab/graphql/tracers/application_context_tracer.rb:23:in `trace'
lib/gitlab/graphql/tracers/timer_tracer.rb:20:in `trace'
lib/gitlab/graphql/tracers/logger_tracer.rb:14:in `trace'
lib/gitlab/graphql/tracers/metrics_tracer.rb:13:in `trace'
lib/gitlab/graphql/tracers/application_context_tracer.rb:23:in `trace'
lib/gitlab/graphql/tracers/timer_tracer.rb:20:in `trace'
lib/gitlab/graphql/tracers/logger_tracer.rb:14:in `trace'
lib/gitlab/graphql/tracers/metrics_tracer.rb:13:in `trace'
lib/gitlab/graphql/tracers/application_context_tracer.rb:23:in `trace'
app/graphql/gitlab_schema.rb:48:in `multiplex'
app/controllers/graphql_controller.rb:212:in `execute_query'
app/controllers/graphql_controller.rb:67:in `execute'
app/controllers/application_controller.rb:520:in `set_current_admin'
lib/gitlab/session.rb:11:in `with_session'
app/controllers/application_controller.rb:511:in `set_session_storage'
lib/gitlab/i18n.rb:107:in `with_locale'
lib/gitlab/i18n.rb:113:in `with_user_locale'
app/controllers/application_controller.rb:502:in `set_locale'
app/controllers/application_controller.rb:495:in `set_current_context'
lib/gitlab/metrics/elasticsearch_rack_middleware.rb:16:in `call'
lib/gitlab/middleware/memory_report.rb:13:in `call'
lib/gitlab/middleware/speedscope.rb:13:in `call'
lib/gitlab/database/load_balancing/rack_middleware.rb:23:in `call'
lib/gitlab/middleware/rails_queue_duration.rb:33:in `call'
lib/gitlab/etag_caching/middleware.rb:21:in `call'
lib/gitlab/metrics/rack_middleware.rb:16:in `block in call'
lib/gitlab/metrics/web_transaction.rb:46:in `run'
lib/gitlab/metrics/rack_middleware.rb:16:in `call'
lib/gitlab/jira/middleware.rb:19:in `call'
lib/gitlab/middleware/go.rb:20:in `call'
lib/gitlab/middleware/query_analyzer.rb:11:in `block in call'
lib/gitlab/database/query_analyzer.rb:37:in `within'
lib/gitlab/middleware/query_analyzer.rb:11:in `call'
lib/gitlab/middleware/multipart.rb:173:in `call'
lib/gitlab/middleware/read_only/controller.rb:50:in `call'
lib/gitlab/middleware/read_only.rb:18:in `call'
lib/gitlab/middleware/same_site_cookies.rb:27:in `call'
lib/gitlab/middleware/basic_health_check.rb:25:in `call'
lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'
lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'
lib/gitlab/middleware/request_context.rb:15:in `call'
lib/gitlab/middleware/webhook_recursion_detection.rb:15:in `call'
config/initializers/fix_local_cache_middleware.rb:11:in `call'
lib/gitlab/middleware/compressed_json.rb:44:in `call'
lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call'
lib/gitlab/middleware/sidekiq_web_static.rb:20:in `call'
lib/gitlab/metrics/requests_rack_middleware.rb:79:in `call'
lib/gitlab/middleware/release_env.rb:13:in `call'
ActiveRecord::DatabaseConnectionError (There is an issue connecting to your database with your username/password, username: gitlab.
Please check your database configuration to ensure the username/password are valid.
):
Just wondering if anyone has seen this before?
