Sidekiq Enqueued queue growing and growing

Hi all,

I have the problem that the Sidekiq Enqueued queue is growing and growing and it seems like Sidekiq does only work on a few jobs every 5 minutes or so. The other queues are quite empty or nearly. From the 25 worker threads only 0-1 is used as shown in top but the sidekiq process always runs with 100% cpu.

I tried to configure a higher concurrency in /opt/gitlab/embedded/service/gitlab-rails/config/sidekiq.yml using
:concurrency: 25
But this doesnt change anything.

Any hint how to get sidekiq to work a bit quicker and more in parallel?

Thanks & have a nice weekend

Basti

Are there fatal errors (non-zero value in the Dead catetegory) in sidekiq, at https://yourgitlab/admin/background_jobs ?

Or just a huge “Enqueued” value but no Errors? What version of gitlab? 8.14.4?

Only one dead job, but quite a lot failed but nothing about them in the logs.
/var/log/gitlab/gitlab-rails/sidekiq.log is empty except “# Logfile created on 2016-03-04 14:18:12 +0100 by logger.rb/44203” and /var/log/gitlab/sidekiq/current only contains information about starts of worker.
Is there a way to tell sidekiq to be a bit more verbose?
TIA

Basti

Might be good to log some more specifics. Use the gitlab rake commands that can report any general system problems with your system. And report your gitlab exact version.

From the above page, I would get this info from your system: (Updated 12/19)

gitlab-rake gitlab:env:info
gitlab-rake gitlab:gitlab_shell:check
gitlab-rake gitlab:sidekiq:check

The Gitlab version is 8.14.3. The output of the check commands

gitlab:env:check

D, [2016-12-19T13:19:51.962636 #24793] DEBUG -- sentry: ** [Raven] Don't know how to build task 'gitlab:env:check' (see --tasks) excluded from capture due to environment or should_capture callback                 
rake aborted!
Don't know how to build task 'gitlab:env:check' (see --tasks)                                                                                                                                                        
/opt/gitlab/embedded/bin/bundle:22:in `load'
/opt/gitlab/embedded/bin/bundle:22:in `<main>'
(See full trace by running task with --trace)  

gitlab:gitlab_shell:check

[all repos ok]

Running /opt/gitlab/embedded/service/gitlab-shell/bin/check                                                                                                                                                          
Ignoring mysql2-0.3.20 because its extensions are not built.  Try: gem pristine mysql2 --version 0.3.20
Ignoring pg-0.19.0 because its extensions are not built.  Try: gem pristine pg --version 0.19.0
Check GitLab API access: OK

Access to /var/opt/gitlab/.ssh/authorized_keys: Ignoring mysql2-0.3.20 because its extensions are not built.  Try: gem pristine mysql2 --version 0.3.20                                                              
Ignoring pg-0.19.0 because its extensions are not built.  Try: gem pristine pg --version 0.19.0                                                                                                                      
OK

Send ping to redis server: OK
gitlab-shell self-check successful

Checking GitLab Shell ... Finished

gitlab:sidekiq:check
Checking Sidekiq …

Running? ... yes
Number of Sidekiq processes ... 1

Checking Sidekiq ... Finished

Ok the first command I should have said:

gitlab-rake gitlab:env:info

You might also want to watch the logs and post any interesting errors, to watch logs its:

gitlabctl-tail

Sidekiq is RUNNING so that’s not the reason your Queue just Grows forever…

gitlab-rake gitlab:env:info

System information
System:         RedHatEnterpriseServer 7.3
Current User:   git
Using RVM:      no
Ruby Version:   2.3.1p112
Gem Version:    2.6.6
Ignoring mysql2-0.3.20 because its extensions are not built.  Try: gem pristine mysql2 --version 0.3.20                                                                                                              
Ignoring pg-0.19.0 because its extensions are not built.  Try: gem pristine pg --version 0.19.0                                                                                                                      
Bundler Version:1.13.6
Rake Version:   10.5.0
Sidekiq Version:4.2.1

GitLab information
Version:        8.14.3
Revision:       2f0065b
Directory:      /opt/gitlab/embedded/service/gitlab-rails

DB Adapter:     postgresql
URL:            https://gitlab.inf.ethz.ch

HTTP Clone URL: https://gitlab.inf.ethz.ch/some-group/some-project.git                                                                                                                                               
SSH Clone URL:  git@gitlab.inf.ethz.ch:some-group/some-project.git                                                                                                                                                   
Using LDAP:     yes
Using Omniauth: yes
                    
Omniauth Providers: google_oauth2, github
GitLab Shell
Version:        4.0.0
Repository storage paths:
- default:      /local/home/git/data/repositories
Hooks:          /opt/gitlab/embedded/service/gitlab-shell/hooks/                                                                                                                                                     
Git:            /opt/gitlab/embedded/bin/git

Sure Sidekiq is running but it had only 1 Thread working on the queue when it should have used 25.
Or is it a wrong information in top that only 1 thread is running?

The logs revealed nothing interessting even no entry for the over 2000 failed Sidekiq jobs. Feels like flying blind and the only way I found to make Sidekiq a bit more verbose is by running it in foreground

/opt/gitlab/embedded/bin/chpst -e /opt/gitlab/etc/gitlab-rails/env -P -U git -u git /opt/gitlab/embedded/bin/bundle exec sidekiq -C /opt/gitlab/embedded/service/gitlab-rails/config/sid
ekiq_queues.yml -e production -r /opt/gitlab/embedded/service/gitlab-rails -t 4 -c 25 -v

Thanks & have a nice day day!

Basti

Oh or is it the GIL in the MRI interpreter that prevents the other 24 threads from running?
But why start 25 threads if only 1 can be running?
Is it not better to start multiple sidekiq processes?