Gitlab slow (13.12.10) (EE License) Need help understanding debug

our gitlab is pretty dam slow at the moment specially on MR pages

8Cores 18GB RAM on a Vcenter VM (average linux load of 1.10)
hosting around 20 projects with only 20 users

All CI pipelines are running on Gitlab runners which we have around 8 gitlab runners

looking at performance monitor we are seeing
500ms/120pg(!) active-record calls: 120 over 100

i increased puma workers from
4 Workers
4 Threads
8 Workers
4 Threads
which has halved the time it takes to load but would like to understand how to fix the warning of below more
500ms/120pg(!) active-record calls: 120 over 100

any help or links would be useful the gitlab page just suggests its hitting a SQL/API limit but no doc on where this limit is set