Hi Team,
I am using GitLab Community Edition 10.3.5.
For CI, i am using the Jenkins server. I have an own webhook integrated with GitLab to trigger the CI process in the Jenkins server for merge requests and note events.
Recently, facing a problem in GitLab that CI triggering for merge request and note events getting delay (i.e. Approximately 1 hour. Sometimes, more than 1 hour). Webhook is not getting the JSON data for the event triggered immediately.
On checking the Sidekiq running processes in GitLab, it was 25 of 25 busy. To resolve this, tried to run the command ‘sudo service gitlab reload’. But this command not working now and getting the message like ‘gitlab: unrecognized service’.
With reference to the SideKiq, we further checked and found that bulk commits doing in the GitLab environment by my colleagues. So, the process_commit count becomes high and suspects which causes the delay in CI triggering. Then modified the automation running time for doing bulk commits to avoid this issue in working hours. Everything went well for 3 more days.
Again the same issue appears. On checking the ProcessCommitWorker logs, it is showing the committed (merged) details which were done before 1 week and before 3 months back.
We also couldn’t find anything related to this issue from the log files such as production.log, api_json.log, application.log, githost.log, grpc.log, production_json.log, repocheck.log, sidekiq.log, gitlab-rails-db- migrate-*.log.
Could anyone please help us with what might be the problem with this issue? How to resolve this issue? Since it spoils our productivity daily.
Thanks in advance.