My Sideque keeps growing and growing all of sudden
My Gitlab version is 14.5.4
busy and enqueue used to be 1 or 0 .
Now suddenly it goes to busy 45 and queue is 30k
and again when not using it goes to 0
when using more it goes to 30k
anyy help is appreciated
Please post on the forums, I am not Gitlab staff I am just community member. Posting on the forum means more people will see your post and you have more chance of getting help.
Sure ,
Posted on Forum only
Pinged you also if any suggestions from you might help
Ah sorry for some reason I thought it was a DM lol my bad.
What do you mean by sidekiq growing? It does queue jobs etc, but maybe you can provide more details as to what is appearing there?
I would suggest upgrading your Gitlab, since there are lots of improvements in the newer versions. There’s a ton of posts on here already addressing upgrade, especially how to understand the upgrade path, and ensuring background migrations are finished before starting the next upgrade. There’s also been a ton of security fixes between your version and the latest, so it can also ensure your Gitlab doesn’t get compromised by doing upgrades regularly.
due to this my jobs are taking time to trigger. it is in queue for more than 10min.
may be after sometime it comes to 0 .
again when there is usage it keeps increasing
any work around
Machine specs? CPU? RAM? Amount of disk free? Also check with top or htop how much CPU is used? Maybe server overloaded, or as I mentioned due to lack of updates, maybe it’s compromised with a CPU Miner that is using all CPU resources. Let us know what the specs are and results of top/htop in CPU usage.
disk is in threshold
CPU is
CPU(s): 48
Worth trying this and increasing the sidekiq processes as well as concurrency: Run multiple Sidekiq processes | GitLab
You should also definitely consider upgrading to the latest which I already mentioned.
Which version will fix this sidekiq any idea
There is no guarantee the upgrade on it’s own will fix it, but it’s recommended for many reasons, features as well as fixes, security fixes to stop your Gitlab server being compromised.
That said, I did link an article about configuring sidekiq processes and concurrency, have you done that? It clearly states on that page:
since you see your jobs piling up you should be looking at reconfiguring your Gitlab based on some of the configuration options listed there that I specifically mentioned - processes and concurrency. That’s more likely to help reduce the queued amount of jobs.
Even if you were not experiencing this problem though, you should be upgrading regularly so that your install is not at risk. It’s enough to search the forum for servers that have been compromised because they weren’t upgraded and ended up having problems with processes dying and crashing because of lack of CPU usage because someone dropped a CPU crypto miner on the machine because of the vulnerabilities that were there. That’s not necessarily saying that yours does or doesn’t have these vulnerabilities since I forget which version it was patched in. However, Gitlab only support the last three versions - which effectively means 15.10, 15.9 and 15.8. Some older versions may get patched if they have critical security vulnerabilities, but Gitlab will only go so far back, but only a few more versions than the ones I mentioned.