Intermittent "ActiveRecord::ConnectionNotEstablished" since upgrade

Problem to solve

We have a self-managed enterprise edition. We upgraded from 18.1 to 18.4 (two steps). The next week we noticed a lot of intermittent Error 500. A count of Error 500 in the NginX logs show almost zero until the day of the upgrade. After that it was around 3000/day.

The applications hitting Gitlab show errors from git pull/fetch/clone. This even occurs intermittently when doing a git clone on a command line so it is not linked to the application. We looked in the gitlab-rails/exceptions_json.log and found error about connection pools. How can we address this?

[details=“ActiveRecord::ConnectionNotEstablished”,“exception.message”:“No connection pool for ‘ActiveRecord::Base’ found.”] {“severity”:“ERROR”,“time”:“2025-10-02T21:49:26.515Z”,“correlation_id”:“01K6KEDJ3BKR2BNRW6464NATQN”,“exception.class”:“ActiveRecord::ConnectionNotEstablished”,“exception.message”:“No connection pool for ‘ActiveRecord::Base’ found.”,“exception.backtrace”:[“activerecord (7.1.5.2) lib/active_record/connection_adapters/abstract/connection_handler.rb:243:in `retrieve_connection’”,“activerecord (7.1.5.2) lib/active_record/connection_handling.rb:287:in `retrieve_connection’”,“lib/feature.rb:24:in `connection’”,“activerecord (7.1.5.2) lib/active_record/model_schema.rb:413:in `table_exists?‘”,“lib/gitlab/database/query_analyzers/prevent_cross_database_modification.rb:52:in `enabled?’”,“lib/gitlab/database/query_analyzer.rb:95:in `block in begin!‘”,“lib/gitlab/database/query_analyzer.rb:92:in `select’”,“lib/gitlab/database/query_analyzer.rb:92:in `begin!’”,“lib/gitlab/database/query_analyzer.rb:80:in `within’”,“lib/gitlab/middleware/query_analyzer.rb:11:in `call’”,“lib/ci/job_token/middleware.rb:11:in `call’”,“batch-loader (2.0.5) lib/batch_loader/middleware.rb:11:in `call’”,“rack-attack (6.7.0) lib/rack/attack.rb:103:in `call’”,“apollo_upload_server (2.1.6) lib/apollo_upload_server/middleware.rb:19:in `call’”,“lib/gitlab/middleware/multipart.rb:174:in `call’”,“rack-attack (6.7.0) lib/rack/attack.rb:127:in `call’”,“warden (1.2.9) lib/warden/manager.rb:36:in `block in call’”,“warden (1.2.9) lib/warden/manager.rb:34:in `catch’”,“warden (1.2.9) lib/warden/manager.rb:34:in `call’”,“rack-cors (2.0.2) lib/rack/cors.rb:102:in `call’”,“rack (2.2.17) lib/rack/tempfile_reaper.rb:15:in `call’”,“rack (2.2.17) lib/rack/etag.rb:27:in `call’”,“rack (2.2.17) lib/rack/conditional_get.rb:27:in `call’”,“rack (2.2.17) lib/rack/head.rb:12:in `call’”,“actionpack (7.1.5.2) lib/action_dispatch/http/permissions_policy.rb:36:in `call’”,“actionpack (7.1.5.2) lib/action_dispatch/http/content_security_policy.rb:36:in `call’”,“lib/gitlab/middleware/read_only/controller.rb:40:in `call’”,“lib/gitlab/middleware/read_only.rb:18:in `call’”,“lib/gitlab/middleware/unauthenticated_session_expiry.rb:18:in `call’”,“rack (2.2.17) lib/rack/session/abstract/id.rb:266:in `context’”,“rack (2.2.17) lib/rack/session/abstract/id.rb:260:in `call’”,“lib/gitlab/middleware/secure_headers.rb:11:in `call’”,“actionpack (7.1.5.2) lib/action_dispatch/middleware/cookies.rb:689:in `call’”,“lib/gitlab/middleware/same_site_cookies.rb:27:in `call’”,“actionpack (7.1.5.2) lib/action_dispatch/middleware/callbacks.rb:29:in `block in call’”,“activesupport (7.1.5.2) lib/active_support/callbacks.rb:101:in `run_callbacks’”,“actionpack (7.1.5.2) lib/action_dispatch/middleware/callbacks.rb:28:in `call’”,“sentry-rails (5.23.0) lib/sentry/rails/rescued_exception_interceptor.rb:14:in `call’”,“actionpack (7.1.5.2) lib/action_dispatch/middleware/debug_exceptions.rb:29:in `call’”,“lib/gitlab/middleware/path_traversal_check.rb:40:in `call’”,“lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call’”,“lib/gitlab/middleware/json_validation.rb:155:in `call’”,“sentry-ruby (5.23.0) lib/sentry/rack/capture_exceptions.rb:30:in `block (2 levels) in call’”,“sentry-ruby (5.23.0) lib/sentry/hub.rb:299:in `with_session_tracking’”,“sentry-ruby (5.23.0) lib/sentry-ruby.rb:428:in `with_session_tracking’”,“sentry-ruby (5.23.0) lib/sentry/rack/capture_exceptions.rb:21:in `block in call’”,“sentry-ruby (5.23.0) lib/sentry/hub.rb:89:in `with_scope’”,“sentry-ruby (5.23.0) lib/sentry-ruby.rb:408:in `with_scope’”,“sentry-ruby (5.23.0) lib/sentry/rack/capture_exceptions.rb:20:in `call’”,“actionpack (7.1.5.2) lib/action_dispatch/middleware/show_exceptions.rb:31:in `call’”,“lib/gitlab/middleware/basic_health_check.rb:25:in `call’”,“lograge (0.11.2) lib/lograge/rails_ext/rack/logger.rb:15:in `call_app’”,“railties (7.1.5.2) lib/rails/rack/logger.rb:24:in `block in call’”,“activesupport (7.1.5.2) lib/active_support/tagged_logging.rb:139:in `block in tagged’”,“activesupport (7.1.5.2) lib/active_support/tagged_logging.rb:39:in `tagged’”,“activesupport (7.1.5.2) lib/active_support/tagged_logging.rb:139:in `tagged’”,“activesupport (7.1.5.2) lib/active_support/broadcast_logger.rb:241:in `method_missing’”,“railties (7.1.5.2) lib/rails/rack/logger.rb:24:in `call’”,“actionpack (7.1.5.2) lib/action_dispatch/middleware/remote_ip.rb:92:in `call’”,“lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call’”,“lib/gitlab/middleware/request_context.rb:15:in `call’”,“lib/gitlab/middleware/webhook_recursion_detection.rb:15:in `call’”,“request_store (1.7.0) lib/request_store/middleware.rb:19:in `call’”,“rack (2.2.17) lib/rack/method_override.rb:24:in `call’”,“rack (2.2.17) lib/rack/runtime.rb:22:in `call’”,“rack-timeout (0.7.0) lib/rack/timeout/core.rb:154:in `block in call’”,“rack-timeout (0.7.0) lib/rack/timeout/support/timeout.rb:19:in `timeout’”,“rack-timeout (0.7.0) lib/rack/timeout/core.rb:153:in `call’”,“config/initializers/fix_local_cache_middleware.rb:11:in `call’”,“lib/gitlab/middleware/compressed_json.rb:44:in `call’”,“actionpack (7.1.5.2) lib/action_dispatch/middleware/executor.rb:14:in `call’”,“lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call’”,“lib/gitlab/metrics/requests_rack_middleware.rb:83:in `call’”,“gitlab-labkit (0.40.0) lib/labkit/middleware/rack.rb:22:in `block in call’”,“gitlab-labkit (0.40.0) lib/labkit/context.rb:36:in `with_context’”,“gitlab-labkit (0.40.0) lib/labkit/middleware/rack.rb:21:in `call’”,“rack (2.2.17) lib/rack/sendfile.rb:110:in `call’”,“actionpack (7.1.5.2) lib/action_dispatch/middleware/request_id.rb:28:in `call’”,“lib/gitlab/middleware/sidekiq_web_static.rb:20:in `call’”,“railties (7.1.5.2) lib/rails/engine.rb:536:in `call’”,“railties (7.1.5.2) lib/rails/railtie.rb:226:in `public_send’”,“railties (7.1.5.2) lib/rails/railtie.rb:226:in `method_missing’”,“lib/gitlab/middleware/release_env.rb:12:in `call’”,“rack (2.2.17) lib/rack/urlmap.rb:74:in `block in call’”,“rack (2.2.17) lib/rack/urlmap.rb:58:in `each’”,“rack (2.2.17) lib/rack/urlmap.rb:58:in `call’”,“puma (6.6.1) lib/puma/configuration.rb:279:in `call’”,“puma (6.6.1) lib/puma/request.rb:99:in `block in handle_request’”,“puma (6.6.1) lib/puma/thread_pool.rb:390:in `with_force_shutdown’”,“puma (6.6.1) lib/puma/request.rb:98:in `handle_request’”,“puma (6.6.1) lib/puma/server.rb:472:in `process_client’”,“puma (6.6.1) lib/puma/server.rb:254:in `block in run’”,“puma (6.6.1) lib/puma/thread_pool.rb:167:in `block in spawn_thread’”],“user.username”:null,“tags.program”:“web”,“tags.locale”:“en”,“tags.feature_category”:null,“tags.correlation_id”:“01K6KEDJ3BKR2BNRW6464NATQN”} [/details]

Steps to reproduce

Upgrade from 18.1.2 to 18.2.6 to 18.4.0

Configuration

Changes attempted to correct this, most done one at a time. None worked.

Server had 4CPU (2 core) and 16GB RAM
puma[‘worker_processes’] = 2
puma[‘min_threads’] = 4
puma[‘max_threads’] = 4
postgresql[‘max_connections’] = 200
sidekiq[‘max_concurrency’] = 50

Server now has 12CPU (6 core) and 32GB RAM. Even tried patch 18.4.1-ee.
puma[‘worker_processes’] = 4
puma[‘min_threads’] = 6
puma[‘max_threads’] = 6
postgresql[‘max_connections’] = 600
sidekiq[‘max_concurrency’] = 70

Versions

Please add an x whether options apply, and add the version information.

• X Self-managed
• GitLab.com SaaS
• Dedicated

Versions

[details=“Now at 18.4.1-ee”]

System information
System:
Proxy: no
Current User: git
Using RVM: no
Ruby Version: 3.2.8
Gem Version: 3.7.1
Bundler Version:2.7.1
Rake Version: 13.0.6
Redis Version: 7.2.10
Sidekiq Version:7.3.9
Go Version: go1.24.6 (Red Hat 1.24.6-1.module+el8.10.0+23407+428597c7) linux/amd64

GitLab information
Version: 18.4.1-ee
Revision: 270836848e7
Directory: /opt/gitlab/embedded/service/gitlab-rails
DB Adapter: PostgreSQL
DB Version: 16.10
URL: https://gitlab.qad.com
HTTP Clone URL: https://gitlab.qad.com/some-group/some-project.git
SSH Clone URL: git@gitlab.qad.com:some-group/some-project.git
Elasticsearch: no
Geo: no
Using LDAP: yes
Using Omniauth: yes
Omniauth Providers: saml

GitLab Shell
Version: 14.45.2
Repository storages:

• default: unix:/var/opt/gitlab/gitaly/gitaly.socket
  GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell

Gitaly

• default Address: unix:/var/opt/gitlab/gitaly/gitaly.socket
• default Version: 18.4.1
• default Git Version: 2.50.1

[/details]

We’ve tried changing the number of postgresql connections, sidekiq concurrency, puma threads and workers… Even increased CPU and RAM. Nothing seems to have helped. This is a single self-managed instance so it never has to leave the box to hit a database. The box does not show any significant load.

I would be tempted to comment out these options by placing a # at the beginning of each line, and let Gitlab manage these settings dynamically and run gitlab-ctl reconfigure. Then see if it performs any better.

Puma normally by default will open a process for each vCPU, so you increased from 4 to 12, but you still limit Puma to 4 processes. Since the 500 errors are for web urls, Nginx isn’t able to forward to the backend server which is Puma.

Once you’ve ruled out any manual changes applied to Gitlab, it can then either be reverted one setting at a time, or tune puma differently with larger settings than what you were using above. This could be a bit of trial and error until you find 500 errors occuring again, then increasing until the 500 errors stop.

Thanks for the suggestions.

I did start with everything commented. Those are the changes I’ve tried. Per your remark I tried increasing the puma threads to match the CPUs but that did not help.

What gets me is the “No connection pool for ‘ActiveRecord::Base’ found”. How would I start running out of connections when the only change was the upgrade?

It’s a good question, my server is 4vcpu and 8gb ram, and the only tuning settings I used to control memory usage was:

puma['worker_processes'] = 2
puma['per_worker_max_memory_mb'] = 1024
sidekiq['concurrency'] = 10

and I’m also using 18.4.1 or rather 18.4.2 today just upgraded. Other than these, no other settings other than external_url and SMTP configuration.

If you are on 18.4.0 it may be worth continuing to 18.4.1 or 18.4.2 of course once you’ve verified that all DB background migrations have finished from the previous upgrades. It could well be something in these releases have fixed something. Assuming you aren’t already on these versions.

The error you are getting above ActiveRecord, I’m just getting generic rails stuff when I google it.

Something else might be worth doing after the upgrade:

gitlab-psql
REINDEX DATABASE;

and see if that helps. Reindexing can help with some performance issues.

All migrations were finished. I did a reindex a few minutes ago and still getting the same errors over and over. Very frustrating.

Can anyone think of something else?

+1 to seeing this type of error with an upgrade to self hosted gitlab 18.4.x CE.

18.3.2 → 18.4.1 on 2025-09-26, we went from 0~2 HTTP 500 errors in nginx/gitlab-access.log a day to 2k~10k/day.
18.4.1 → 18.4.2 on 2025-10-08, the count came down to 2k~5k/day. So not sure if that’s “better.”

Correspondingly, we saw “No connection pool for ‘ActiveRecord::Base’ found.” hits in gitlab-rails/exceptions.json.log go from 0 to 13k~33k/day after going to 18.4.1 and then 6k~10k/day after going to 18.4.2.

There was one day of exception from the above patterns where a script we were using to scrape the API wasn’t running on a Saturday (when end user load is also low) and we saw merely 51 HTTP-500 errors and 295 “No connection pool” errors that day during the 18.4.2 run.

We have not changed any puma settings related to processes or threads in gitlab.rb, but see /var/opt/gitlab/gitlab-rails/etc/puma.rb has:
threads 4, 4
workers 4

in /opt/gitlab/embedded/nodes/${my_servers_fqdn}.json
{
“normal”: {
“gitlab”: {
“sidekiq”: {
“concurrency”: 20,
…

and /var/opt/gitlab/postgresql/data/postgresql.conf has
max_connections = 400

which I presume to be defaults.

2 virtual CPUs each with 2 cores = 4 total cores.
16 GiB RAM.

We get north of 2.5 million hits a day average and have been running that script for a while before going to 18.4. Postgres and such are the embedded versions.

The issue described on "No connection pool for 'ActiveRecord::Base' found." every 5 minutes after 18.3 -> 18.4 update (#573581) · Issues · GitLab.org / GitLab · GitLab sounds plausibly like it may be this issue.

I applied the lines from !208836 and my problems went away. Thanks for the heads up on that issue!