Sidekiq Log : "ActiveRecord::RecordNotFound: Couldn't find Member with 'id' " Activity and tags not visible


I see this error in sidekiq log

2016-08-29_12:12:53.53935 2016-08-29T12:12:53.539Z 15598 TID-uyd1w ActionMailer::DeliveryJob JID-c84389d449f68edc9585e298 INFO: fail: 0.015 sec
2016-08-29_12:12:53.53960 2016-08-29T12:12:53.539Z 15598 TID-uyd1w WARN: {"class":"ActiveJob::QueueAdapters::SidekiqAdapter::JobWrapper","wrapped":"ActionMailer::DeliveryJob","queue":"mailers","args":[{"job_class":"ActionMailer::DeliveryJob","job_id":"06b8a6bf-83c6-423a-b13d-06a8121ecaa6","queue_name":"mailers","arguments":["Notify","member_access_granted_email","deliver_now","Project",13964],"locale":"en"}],"retry":true,"jid":"c84389d449f68edc9585e298","created_at":1472472773.5230498,"enqueued_at":1472472773.5231678,"error_message":"Couldn't find Member with 'id'=13964","error_class":"ActiveRecord::RecordNotFound","failed_at":1472472773.5326986,"retry_count":0}
2016-08-29_12:12:53.53969 2016-08-29T12:12:53.539Z 15598 TID-uyd1w WARN: ActiveRecord::RecordNotFound: Couldn't find Member with 'id'=13964
2016-08-29_12:12:53.53981 2016-08-29T12:12:53.539Z 15598 TID-uyd1w WARN: /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.2.6/lib/active_record/relation/finder_methods.rb:324:in `raise_record_not_found_exception!'

some times this also (sidekiq also reboots)

2016-08-29_14:43:16.20367 2016-08-29T14:43:16.203Z 15598 TID-c61yc ProjectCacheWorker JID-1f343cbb6b49cb92b4cbdae2 INFO: done: 0.251 sec
2016-08-29_14:43:41.98576 2016-08-29T14:43:41.985Z 15598 TID-12a7iw WARN: sending SIGTERM to PID 15598 - Worker PostReceive - JID-45dd7615d39101eba163f995
2016-08-29_14:43:41.98580 2016-08-29T14:43:41.985Z 15598 TID-12a7iw WARN: waiting 30 seconds before sending SIGKILL to PID 15598 - Worker PostReceive - JID-45dd7615d39101eba163f995
2016-08-29_14:43:41.98700 2016-08-29T14:43:41.986Z 15598 TID-awjbs INFO: Shutting down
2016-08-29_14:43:41.98726 2016-08-29T14:43:41.987Z 15598 TID-c60lg INFO: Scheduler exiting...
2016-08-29_14:43:41.98747 2016-08-29T14:43:41.987Z 15598 TID-awjbs INFO: Terminating quiet workers
2016-08-29_14:43:41.98764 2016-08-29T14:43:41.987Z 15598 TID-13o0m0 INFO: Scheduler exiting...
2016-08-29_14:43:42.48810 2016-08-29T14:43:42.487Z 15598 TID-awjbs INFO: Pausing to allow workers to finish...
2016-08-29_14:43:44.99033 2016-08-29T14:43:44.990Z 15598 TID-awjbs INFO: Bye!
2016-08-29_14:44:05.46307 2016-08-29T14:44:05.462Z 13175 TID-6je10 INFO: Booting Sidekiq 4.1.2 with redis options {:url=>"unix:/var/opt/gitlab/redis/redis.socket", :namespace=>"resque:gitlab"}

I was seeing failed and dead jobs in Web UI (admin->Monitoring->BackgroundJobs), i cleared the dead jobs.

Because of this problem frequently Activity and Tags are not visible , I have to kill sidekiq and restart Gitlab.

My system info:

System:         RedHatEnterpriseServer 6.6
Current User:   git
Using RVM:      no
Ruby Version:   2.1.8p440
Gem Version:    2.5.1
Bundler Version:1.10.6
Rake Version:   10.5.0
Sidekiq Version:4.1.2

GitLab information
Version:        8.9.6
Revision:       ba3d01c
Directory:      /opt/gitlab/embedded/service/gitlab-rails
DB Adapter:     postgresql
SSH Clone URL:
Using LDAP:     yes
Using Omniauth: no

GitLab Shell
Version:        3.0.0
Repositories:   /var/opt/gitlab/git-data/repositories
Hooks:          /opt/gitlab/embedded/service/gitlab-shell/hooks/
Git:            /opt/gitlab/embedded/bin/git

Has any one faced similar issue is there any solution to this ? Any clues are appreciated.

copy: @stanhu @axil @kelvinmutuma

Hi @ershad.ahmad is the error always the same? Looking for a member with the same id (13964)?


the same here:

System information
System:         Debian 8.9
Current User:   git
Using RVM:      no
Ruby Version:   2.3.3p222
Gem Version:    2.6.6
Bundler Version:1.13.7
Rake Version:   12.0.0
Redis Version:  3.2.5
Git Version:    2.13.5
Sidekiq Version:5.0.4
Go Version:     unknown

GitLab information
Version:        9.5.4
Revision:       fbffc27
Directory:      /opt/gitlab/embedded/service/gitlab-rails
DB Adapter:     postgresql
SSH Clone URL:
Using LDAP:     yes
Using Omniauth: no

GitLab Shell
Version:        5.8.0
Repository storage paths:
- default:      /var/opt/gitlab/git-data/repositories
Hooks:          /opt/gitlab/embedded/service/gitlab-shell/hooks
Git:            /opt/gitlab/embedded/bin/git

Actually there are problems deleting projects (project deletion pending but never really deleted) but I don’t know if this is related.

Where to look next?

This issue we had in 8.9.6 after we upgrade to higher version (8.14.6) the issue was resolved. Till we upgrade we ran a script to restart the sidekiq process every 6 hours.