Sidekip problems : queue growing, merge requesting or forking too long

How to Use GitLab
1

Hello,

Sorry for this long title but we have a “sidekiq problem” impacting differents gitlab functionalities.

Gitlab CE version installed : 11.1.4
with an external postgreSQL server : 9.6.8 (server 9.6.3)
OS: Red Hat Enterprise Linux Server release 7.2 (Maipo)

image
image832×155 11.3 KB

image

At start, every thing looks good and works fine.

But one or two hours after starting gitlab services:

  • The number of sidekig threads displayed on the UI is growing with, at my opinion, no reason till 25 of 25 busy…
  • In parallel, the enqueued background jobs is also growing (more than 640 now)…
  • A merge request or a project fork is taking between 20 minutes and 2 hours (the message “This merge request is in the process of being merged” is displayed during this time). The project used contains 3 simples files. The merge request concerns only a single file within one or two lines…

All of this, with nearly no gitlab activity. I mean, I’m the only one to used it as it’s a new instance we test before openning the service. And I did nothing else than one or two commits, 5 merges requests and 2 or 3 fork :frowning:

the gitlab status or rake commands returns looks good to me (no error).
What else can I check / do ?

Thank you for your advice.

See below screen shots and logs.

Merge request example:
image

Sidekiq logs:
…/…
2018-09-07_12:09:46.30008 2018-09-07T12:09:46.300Z 782 TID-oubbfnb8q PagesDomainVerificationCronWorker JID-4e4f7111a6ccbeb16a66cdb6 INFO: start
2018-09-07_12:09:46.30894 2018-09-07T12:09:46.308Z 782 TID-oubbj08xm Ci::ArchiveTracesCronWorker JID-92f3b29b1092d5ea6576e7a1 INFO: done: 8354.312 sec
2018-09-07_12:09:46.30942 2018-09-07T12:09:46.308Z 782 TID-oubbj08ge PagesDomainVerificationCronWorker JID-17faa111d400d5631d365b58 INFO: done: 11139.007 sec
2018-09-07_12:09:46.30951 2018-09-07T12:09:46.309Z 782 TID-oubbj07su RepositoryCheck::DispatchWorker JID-a005188860cbaf38d4a95a38 INFO: done: 7426.056 sec
2018-09-07_12:09:46.31029 2018-09-07T12:09:46.310Z 782 TID-oubbj08xm ImportExportProjectCleanupWorker JID-c975b9fbbc4437ab8a6dceb5 INFO: start
2018-09-07_12:09:46.31093 2018-09-07T12:09:46.310Z 782 TID-oube6gmsi PagesDomainVerificationCronWorker JID-0a5aa4ae41e2886e7445143c INFO: done: 6497.802 sec
2018-09-07_12:09:46.31136 2018-09-07T12:09:46.311Z 782 TID-oube6gmsi PagesDomainVerificationCronWorker JID-3dc9597ed1739eee0c865bbc INFO: start
…/…

Sidekiq dashboard:

image
image1040×812 68.1 KB

2

I found a solution by setting environment variables specific to GEM Memory Killer.

As read on the Sidekiq MemoryKiller page, I add the following lines to my configuration file “gitlab.rb” then launch a gitlab-ctl reconfigure et voilà!

gitlab_rails[‘env’] = {
‘SIDEKIQ_MEMORY_KILLER_MAX_RSS’ => “1000000”,
‘SIDEKIQ_MEMORY_KILLER_GRACE_TIME’ => “900”,
‘SIDEKIQ_MEMORY_KILLER_SHUTDOWN_WAIT’ => “30”
}

The first variable was already configured by the gitlab configuration template (not shown in gitlab.rb at first) I have kept the values indicated on the administration page mentioned above.