GitLab version 16.5.1
Docker deployment
Core i7, 8 cpu cores
I saw strange behavior that the CPU load was around 50% when GitLab was idle (without any activity)
I see this in running tasks MergeRequestCleanupRefsWorker
Tell me, what is MergeRequestCleanupRefsWorker job, where I can read the doc about it? And I think that it is precisely these tasks that load the CPU, how to deal with this? Is it possible to optimize it somehow?
When i stop this jobs, they rerun again after 1 minute
I have the same problem. Yesterday evening I updated the gitlab version 16.3.5 to 16.5.1. I checked all the configs according to the official documentation and found no errors. But within 2 hours the MergeRequestCleanupRefsWorker jobs are running and the overall CPU usage is around 80-85%. Stopping the jobs helps for about one minute, after which the jobs start again.
As a result, in 2 hours in the history of background jobs, over 70 thousand jobs were processed. Which is 2.5 times more per day on our server.
In earlier versions the CPU load was about 15%
GitLab CE version 16.5.1
Centos 7
4 cpu cores, 16 gb ram
@Antom91 , ัan you tell me if you solved this problem?
Not so, after 4 hours this jobs MergeRequestCleanupRefsWorker was ended. I think it depends on speed of your HHD\SSD
Hi, @Antom91
Thanks for the answer.
Yes, the tasks have ended, but the number of tasks in the Background Jobs is growing. Now 60-70 thousand entries per day, previously there were about 10 thousand
We ran into the same problem and discovered the issue:
Way back in 14.1, the way the MergeRequestCleanupRefsWorker is scheduled was changed - Improve the way MergeRequestCleanupRefsWorker gets scheduled (#296874) ยท Issues ยท GitLab.org / GitLab ยท GitLab
This new scheduling was placed behind the merge_request_refs_cleanup feature flag: [Feature flag] Enable merge_request_refs_cleanup feature flag (#336070) ยท Issues ยท GitLab.org / GitLab ยท GitLab
The flag was globally enabled in 16.4: Set merge_request_refs_cleanup default_enabled: true (!130098) ยท Merge requests ยท GitLab.org / GitLab ยท GitLab, which is why youโre seeing the new behavior unexpectedly as of 16.6
The flag has been removed in 16.6: Remove merge_request_refs_cleanup feature flag (!130004) ยท Merge requests ยท GitLab.org / GitLab ยท GitLab, so we need to request that the flag is reintroduced or something given the performance regression it carries
If youโre still on 16.5.x, you can disable the merge_request_refs_cleanup to return to the previous behavior
After exactly 3 hours of insane load, the MergeRequestCleanupRefsWorker tasks are finished and everything works fine.
