Hi,
My gitlab instance is already overutilized as like below,
And facing severe performance issue such as enqueued jobs more than 25K which is making delay in all execution in sidekiq.
Could anyone can help me to fine tune the current environment for better performance.
Is there any way to resolve this?
i believe you need first to work with a sys admin that can investigate where are the bottlenecks in that machine resources (CPU, RAM, IO) correlating with the user activity
-d
There is an option to increase the RAM to 64 and increase the sidekiq workers to 70 or above. Is that helps ? How about improving kernal parameters proportional to gitlab performance improvement ?
Any suggestion?
@darianf01 already wrote that it’s something you need to figure out, possibly in collaboration with a sys admin that has access to your system. We can’t help you any more.
@palakkadansar
Can you provide additional information about the system?
What are you using for disk/storage? Are you using SSD or HDD? What you you using for mounted volumes and file system?
What workloads and CI/CD pipelines are causing so many en-queued jobs?
Is this a recent issue or an ongoing problem?
You should see a significant boost in performance by upgrading to a newer version.
Updating to the latest version even provides build-in Grafana dashboards which can help in visualizing bottlenecks.
Hi,
Please find the details below,
Hello @palakkadansar,
Are you running GitLab Runner on the same machine you’re running Gitlab on?
Depending on how you decide to configure GitLab Runner and what tools you use in your build hooks and pipelines, GitLab Runner can consume a significant amount of available memory. We recommend using a separate machine for GitLab Runner.
If your Runner is currently running on a separate machine, the bottleneck is likely related to IO limits of your HDD.
You can gather and share filesystem benchmarking data to investigate further.
In any case, migrating to SSD storage should improve performance significantly. 
Upgrading your GitLab version should also bring some performance improvements.
We currently using runner as separate physical machine. we use two shared runner (docker) separately and we have 3K specific runner as well.
Sorry for wrong info , Currently we are using SSD only
The storage bench marking already done and we are getting good performance as well.
Up-gradation is planned but it might take delay for 1 more month before we are trying to find some solution.