GitLab CPU usage by sidekiq over 50%

GitLab version 16.5.1
Docker deployment
Core i7, 8 cpu cores

I saw strange behavior that the CPU load was around 50% when GitLab was idle (without any activity)

I see this in running tasks MergeRequestCleanupRefsWorker

Tell me, what is MergeRequestCleanupRefsWorker job, where I can read the doc about it? And I think that it is precisely these tasks that load the CPU, how to deal with this? Is it possible to optimize it somehow?

When i stop this jobs, they rerun again after 1 minute

2 Likes

I have the same problem. Yesterday evening I updated the gitlab version 16.3.5 to 16.5.1. I checked all the configs according to the official documentation and found no errors. But within 2 hours the MergeRequestCleanupRefsWorker jobs are running and the overall CPU usage is around 80-85%. Stopping the jobs helps for about one minute, after which the jobs start again.

As a result, in 2 hours in the history of background jobs, over 70 thousand jobs were processed. Which is 2.5 times more per day on our server.

In earlier versions the CPU load was about 15%

GitLab CE version 16.5.1
Centos 7
4 cpu cores, 16 gb ram

@Antom91 , сan you tell me if you solved this problem?

1 Like

Not so, after 4 hours this jobs MergeRequestCleanupRefsWorker was ended. I think it depends on speed of your HHD\SSD

2 Likes

Hi, @Antom91
Thanks for the answer.
Yes, the tasks have ended, but the number of tasks in the Background Jobs is growing. Now 60-70 thousand entries per day, previously there were about 10 thousand

1 Like

Hey @Alex23 @Antom91,

We ran into the same problem and discovered the issue:

Way back in 14.1, the way the MergeRequestCleanupRefsWorker is scheduled was changed - Improve the way MergeRequestCleanupRefsWorker gets scheduled (#296874) · Issues · GitLab.org / GitLab · GitLab

This new scheduling was placed behind the merge_request_refs_cleanup feature flag: [Feature flag] Enable merge_request_refs_cleanup feature flag (#336070) · Issues · GitLab.org / GitLab · GitLab

The flag was globally enabled in 16.4: Set merge_request_refs_cleanup default_enabled: true (!130098) · Merge requests · GitLab.org / GitLab · GitLab, which is why you’re seeing the new behavior unexpectedly as of 16.6

The flag has been removed in 16.6: Remove merge_request_refs_cleanup feature flag (!130004) · Merge requests · GitLab.org / GitLab · GitLab, so we need to request that the flag is reintroduced or something given the performance regression it carries

If you’re still on 16.5.x, you can disable the merge_request_refs_cleanup to return to the previous behavior

2 Likes

After exactly 3 hours of insane load, the MergeRequestCleanupRefsWorker tasks are finished and everything works fine.