Cant merge after upgrade (sidekiq related ?)

StephaneK · July 20, 2022, 12:23pm

Hi, yesterday I migrated from a 13.XX to the last 15.X.X version using the upgrade path and ubuntu packages.
Today my team is complaining that they can’t merge branches anymore, the error message is
“The source branch XXX does not exist. Please restore it or use a different source branch.”

I can see some error with sidekiq using the command : gitlab-rake gitlab:check

running the suggested background_job command from /opt/gitlab/embedded/service/gitlab-rails
produce this error :

Usage: RAILS_ENV=<env> SIDEKIQ_WORKERS=<n> bin/background_jobs {stop|start|start_foreground|restart}
No pidfile found at /opt/gitlab/embedded/service/gitlab-rails/tmp/pids/sidekiq-cluster.pid; is Sidekiq running?

It also seems that there are background migrations stuck at 99%

iwalker · July 20, 2022, 12:26pm

What is output of:

gitlab-ctl status

StephaneK · July 20, 2022, 12:33pm

gitlab-ctl status
run: alertmanager: (pid 5064) 5347s; run: log: (pid 5586) 16569178s
run: gitaly: (pid 5092) 5347s; run: log: (pid 5570) 16569178s
run: gitlab-exporter: (pid 5117) 5347s; run: log: (pid 5587) 16569178s
run: gitlab-kas: (pid 5131) 5346s; run: log: (pid 10212) 82850s
run: gitlab-workhorse: (pid 5159) 5346s; run: log: (pid 5555) 16569178s
run: logrotate: (pid 9137) 1745s; run: log: (pid 5585) 16569178s
run: nginx: (pid 5186) 5345s; run: log: (pid 5557) 16569178s
run: postgres-exporter: (pid 5214) 5345s; run: log: (pid 5562) 16569178s
run: postgresql: (pid 5223) 5343s; run: log: (pid 5569) 16569178s
run: puma: (pid 5231) 5342s; run: log: (pid 5580) 16569178s
run: redis: (pid 5238) 5341s; run: log: (pid 5590) 16569178s
run: redis-exporter: (pid 5244) 5340s; run: log: (pid 5589) 16569178s
run: registry: (pid 5253) 5340s; run: log: (pid 5588) 16569178s
run: sidekiq: (pid 11676) 10s; run: log: (pid 5561) 16569178s

it seems the sidekiq process is restarting ?

iwalker · July 20, 2022, 12:37pm

I would try, as usually when problems occur after upgrades, suggested is:

gitlab-ctl reconfigure
gitlab-ctl restart

and if still problematic:

systemctl restart gitlab-runsvdir

somtimes gitlab-ctl restart doesn’t restart everything, wherease the systemctl restart will actually ensure that any leftover processes that weren’t restarted, will be sorted out.

Try the rake check command after this and see if it’s any better.

iwalker · July 20, 2022, 12:40pm

Also for the sudo command you used, to be honest I find it better to change to root user, and run the commands without sudo. Otherwise you will need to use -E parameter to pass through variables as they don’t get passed otherwise. So:

sudo -E

and then the rest of the sudo command. The same for gitlab-rails, instead of sudo, change to root with su and then run it.

StephaneK · July 20, 2022, 12:48pm

I tried the reconfigure / restart and then systemctl restart, still shows the same error after the rake check
with the sudo -E I am getting a slightly different error now:

sudo -E -u git -H RAILS_ENV=production bin/background_jobs start

No pidfile found at /opt/gitlab/embedded/service/gitlab-rails/tmp/pids/sidekiq-cluster.pid; is Sidekiq running?
tee: /opt/gitlab/embedded/service/gitlab-rails/log/sidekiq.log: Permission denied
/opt/gitlab/embedded/service/gitlab-rails# /usr/lib/ruby/2.5.0/rubygems/core_ext/kernel_require.rb:59:in `require': cannot load such file -- bundler (LoadError)
        from /usr/lib/ruby/2.5.0/rubygems/core_ext/kernel_require.rb:59:in `require'
        from /opt/gitlab/embedded/service/gitlab-rails/config/bundler_setup.rb:9:in `<top (required)>'
        from /opt/gitlab/embedded/service/gitlab-rails/sidekiq_cluster/cli.rb:3:in `require_relative'
        from /opt/gitlab/embedded/service/gitlab-rails/sidekiq_cluster/cli.rb:3:in `<top (required)>'
        from /opt/gitlab/embedded/service/gitlab-rails/bin/sidekiq-cluster:4:in `require_relative'
        from /opt/gitlab/embedded/service/gitlab-rails/bin/sidekiq-cluster:4:in `<main>'

iwalker · July 20, 2022, 12:50pm

Try from standard user with su to change to root:

sudo su

then when active as root user, try the rake/rails command without sudo so natively like:

gitlab-rake gitlab:check

I find more success this way than using sudo. far less problems.

StephaneK · July 20, 2022, 12:55pm

yes I did all previous command with the standar root user (no sudo)
except the one from background_jobs, if I run the background_job as root i am getting the same error as with -E

No pidfile found at /opt/gitlab/embedded/service/gitlab-rails/tmp/pids/sidekiq-cluster.pid; is Sidekiq running?

and then

/usr/lib/ruby/2.5.0/rubygems/core_ext/kernel_require.rb:59:in `require': cannot load such file -- bundler (LoadError)
        from /usr/lib/ruby/2.5.0/rubygems/core_ext/kernel_require.rb:59:in `require'
        from /opt/gitlab/embedded/service/gitlab-rails/config/bundler_setup.rb:9:in `<top (required)>'
        from /opt/gitlab/embedded/service/gitlab-rails/sidekiq_cluster/cli.rb:3:in `require_relative'
        from /opt/gitlab/embedded/service/gitlab-rails/sidekiq_cluster/cli.rb:3:in `<top (required)>'
        from /opt/gitlab/embedded/service/gitlab-rails/bin/sidekiq-cluster:4:in `require_relative'
        from /opt/gitlab/embedded/service/gitlab-rails/bin/sidekiq-cluster:4:in `<main>'

does it means some file is missing or something, I don;t know ruby

iwalker · July 20, 2022, 12:57pm

Well, if you find bundler like this:

cd /opt/gitlab
find ./ -iname bundler

./embedded/bin/bundler
./embedded/lib/ruby/site_ruby/2.7.0/bundler
./embedded/lib/ruby/2.7.0/bundler
./embedded/lib/ruby/gems/2.7.0/gems/bundler-2.1.4/exe/bundler
./embedded/lib/ruby/gems/2.7.0/gems/bundler-2.3.15/exe/bundler
./embedded/lib/ruby/gems/2.7.0/gems/bundler-2.3.15/lib/bundler

then it means bundler exists. I’m curious if your results will be the same as mine. This is from Gitlab 15.1.3 though.

The pid for sidekiq never exists anyway, the location is a symlink to /var/opt/gitlab/gitlab-rails/tmp and it’s empty for me as well, and works. So something else is not right with your system after the upgrade. In between each upgrade on the path, did you wait for all background migrations to finish first before continuing the next upgrade?

I would use this command from the console as well:

gitlab-rake db:migrate:status

StephaneK · July 20, 2022, 1:51pm

sorry for the delay, I found the bundler with your command but it seems the version of ruby is different (2.7.0) compare to what the command what looking for (2.5.0).

find ./ -iname bundler
./embedded/lib/ruby/2.7.0/bundler
./embedded/lib/ruby/site_ruby/2.7.0/bundler
./embedded/lib/ruby/gems/2.7.0/gems/bundler-2.3.15/lib/bundler
./embedded/lib/ruby/gems/2.7.0/gems/bundler-2.3.15/exe/bundler
./embedded/lib/ruby/gems/2.7.0/gems/bundler-2.1.4/exe/bundler
./embedded/bin/bundler

How could I set it to look for 2.7 instead of 2.5

concerning the migrations everything seems to be showing up

iwalker · July 20, 2022, 2:33pm

You got any other ruby packages installed on your system?

        from /usr/lib/ruby/2.5.0/rubygems/core_ext/kernel_require.rb:59:in `require'

that was in your error. Do:

dpkg -l | grep -i ruby

If you have, then you have a conflict between what Gitlab has installed, and what is on your system. Are you using this server for something else as well as Gitlab? If so, then I suggest that you run Gitlab on it’s own server, so either move it, or remove the ruby packages installed from the Ubuntu repositories.

Gitlab should only be finding stuff under /opt/gitlab - so since it found something system-wide that is causing the problem.

StephaneK · July 20, 2022, 2:46pm

 dpkg -l | grep -i ruby
ii  libruby2.5:amd64                          2.5.1-1ubuntu1.12                               amd64        Libraries necessary to run Ruby 2.5
ii  rake                                      12.3.1-1ubuntu0.1                               all          ruby make-like utility
ii  ruby                                      1:2.5.1                                         amd64        Interpreter of object-oriented scripting language Ruby (default version)
ii  ruby-did-you-mean                         1.2.0-2                                         all          smart error messages for Ruby > 2.3
ii  ruby-minitest                             5.10.3-1                                        all          Ruby test tools supporting TDD, BDD, mocking, and benchmarking
ii  ruby-net-telnet                           0.1.1-2                                         all          telnet client library
ii  ruby-power-assert                         0.3.0-1                                         all          library showing values of variables and method calls in an expression
ii  ruby-test-unit                            3.2.5-1                                         all          unit testing framework for Ruby
ii  ruby2.5                                   2.5.1-1ubuntu1.12                               amd64        Interpreter of object-oriented scripting language Ruby
ii  rubygems-integration                      1.11                                            all          integration of Debian Ruby packages with Rubygems

doesn’t look like anything serious is using ruby, it was probably from old gitlab install, I will try to uninstall first.

StephaneK · July 20, 2022, 3:08pm

after removing all 2.5 ruby I still get the first part of the message about the “No pidfile found”
So search through the sidekiq logs /var/log/gitlab/sidekiq/current
and It seems to get those ligne every minutes

[2022-07-20 16:03:06] INFO  WEBrick 1.6.1
{"severity":"INFO","time":"2022-07-20T15:03:06.822Z","class":"Gitlab::SidekiqDaemon::MemoryKiller","action":"start","pid":28626,"message":"Starting Gitlab::SidekiqDaemon::MemoryKiller Daemon","retry":0}      
[2022-07-20 16:03:06] INFO  ruby 2.7.5 (2021-11-24) [x86_64-linux]
{"severity":"WARN","time":"2022-07-20T15:03:06.854Z","class":"Gitlab::SidekiqDaemon::MemoryKiller","action":"stop","pid":28626,"message":"Stopping Gitlab::SidekiqDaemon::MemoryKiller Daemon","retry":0}
{"severity":"INFO","time":"2022-07-20T15:03:09.861Z","message":"A worker terminated, shutting down the cluster"}
{"severity":"INFO","time":"2022-07-20T15:03:16.450Z","message":"Starting cluster with 1 processes"}

is it normal expected behavior ? what is the Memorykiller thing

iwalker · July 20, 2022, 3:32pm

What are your cpu/ram specs of this machine? Gitlab should have 8gb to run properly, and 4cpu. Any less and it can be problematic.

After removing the ruby stuff, suggest restarting the server completely to make sure gitlab starts cleanly. Could have been running processes with the wrong version of ruby.

StephaneK · July 20, 2022, 3:37pm

the server should have enough ram/cpu the server has 24cores 64gb or ram
I tried disabling Memorykiller using an env variable in the config file + reconfigure

gitlab_rails['env'] = {
    'SIDEKIQ_MEMORY_KILLER_MAX_RSS' => "0"
}

Memorykiller is not showing up in the log anymore but the sidekiq still seems to be constantly rebooting

{"severity":"INFO","time":"2022-07-20T15:33:33.296Z","message":"Cron Jobs - add job with name: geo_sidekiq_cron_config_worker"}
{"severity":"WARN","time":"2022-07-20T15:33:36.022Z","message":"Creating scope :without_statuses. Overwriting existing method Ci::Pipeline.without_statuses."}
{"severity":"WARN","time":"2022-07-20T15:33:37.134Z","message":"Creating scope :of_projects. Overwriting existing method MergeRequest.of_projects."}
{"severity":"WARN","time":"2022-07-20T15:33:37.136Z","message":"Creating scope :join_project. Overwriting existing method MergeRequest.join_project."}
{"severity":"WARN","time":"2022-07-20T15:33:37.136Z","message":"Creating scope :references_project. Overwriting existing method MergeRequest.references_project."}
{"severity":"WARN","time":"2022-07-20T15:33:37.217Z","message":"Creating scope :opened. Overwriting existing method Epic.opened."}
{"severity":"WARN","time":"2022-07-20T15:33:37.217Z","message":"Creating scope :closed. Overwriting existing method Epic.closed."}
{"severity":"WARN","time":"2022-07-20T15:33:37.238Z","message":"Creating scope :with_api_entity_associations. Overwriting existing method Epic.with_api_entity_associations."}
{"severity":"WARN","time":"2022-07-20T15:33:37.552Z","message":"Creating scope :system. Overwriting existing method Note.system."}
{"severity":"WARN","time":"2022-07-20T15:33:37.949Z","message":"Creating scope :order_created_desc. Overwriting existing method Packages::Package.order_created_desc."}
{"severity":"WARN","time":"2022-07-20T15:33:37.949Z","message":"Creating scope :order_name_desc. Overwriting existing method Packages::Package.order_name_desc."}
{"severity":"WARN","time":"2022-07-20T15:33:38.412Z","message":"Creating scope :closed. Overwriting existing method Iteration.closed."}
{"severity":"WARN","time":"2022-07-20T15:33:38.413Z","message":"Creating scope :within_timeframe. Overwriting existing method Iteration.within_timeframe."}
{"severity":"WARN","time":"2022-07-20T15:33:41.533Z","message":"Creating scope :open. Overwriting existing method AlertManagement::Alert.open."}
{"severity":"WARN","time":"2022-07-20T15:33:41.596Z","message":"Creating scope :open. Overwriting existing method IncidentManagement::IssuableEscalationStatus.open."}
{"severity":"WARN","time":"2022-07-20T15:33:41.744Z","message":"Creating scope :with_associations. Overwriting existing method Board.with_associations."}
{"severity":"WARN","time":"2022-07-20T15:33:41.910Z","message":"Creating scope :movable. Overwriting existing method Boards::EpicList.movable."}
{"severity":"WARN","time":"2022-07-20T15:33:42.869Z","message":"Creating scope :order_name_asc. Overwriting existing method Label.order_name_asc."}
{"severity":"WARN","time":"2022-07-20T15:33:42.870Z","message":"Creating scope :order_name_desc. Overwriting existing method Label.order_name_desc."}
{"severity":"WARN","time":"2022-07-20T15:33:43.920Z","message":"Creating scope :open. Overwriting existing method ExternalPullRequest.open."}
{"severity":"WARN","time":"2022-07-20T15:33:44.146Z","message":"Creating scope :order_name_asc. Overwriting existing method Member.order_name_asc."}
{"severity":"WARN","time":"2022-07-20T15:33:44.146Z","message":"Creating scope :order_name_desc. Overwriting existing method Member.order_name_desc."}
{"severity":"WARN","time":"2022-07-20T15:33:44.329Z","message":"Creating scope :with_logging_info. Overwriting existing method PagesDomain.with_logging_info."}
{"severity":"WARN","time":"2022-07-20T15:33:44.619Z","message":"Creating scope :system. Overwriting existing method PrometheusMetric.system."}
{"severity":"WARN","time":"2022-07-20T15:33:45.176Z","message":"Creating scope :with_user. Overwriting existing method GroupMember.with_user."}
{"severity":"WARN","time":"2022-07-20T15:33:45.178Z","message":"Creating scope :reporters. Overwriting existing method GroupMember.reporters."}
{"severity":"WARN","time":"2022-07-20T15:33:45.179Z","message":"Creating scope :guests. Overwriting existing method GroupMember.guests."}
{"severity":"WARN","time":"2022-07-20T15:33:45.549Z","message":"Creating scope :extended. Overwriting existing method GitlabSubscription.extended."}
{"severity":"WARN","time":"2022-07-20T15:33:46.363Z","message":"Creating scope :without_statuses. Overwriting existing method Ci::Stage.without_statuses."}
{"severity":"WARN","time":"2022-07-20T15:33:46.364Z","message":"Creating scope :created. Overwriting existing method Ci::Stage.created."}
{"severity":"WARN","time":"2022-07-20T15:33:46.365Z","message":"Creating scope :pending. Overwriting existing method Ci::Stage.pending."}
{"severity":"WARN","time":"2022-07-20T15:33:46.365Z","message":"Creating scope :running. Overwriting existing method Ci::Stage.running."}
{"severity":"WARN","time":"2022-07-20T15:33:46.365Z","message":"Creating scope :success. Overwriting existing method Ci::Stage.success."}
{"severity":"WARN","time":"2022-07-20T15:33:46.366Z","message":"Creating scope :failed. Overwriting existing method Ci::Stage.failed."}
{"severity":"WARN","time":"2022-07-20T15:33:46.366Z","message":"Creating scope :canceled. Overwriting existing method Ci::Stage.canceled."}
{"severity":"WARN","time":"2022-07-20T15:33:46.366Z","message":"Creating scope :skipped. Overwriting existing method Ci::Stage.skipped."}
{"severity":"WARN","time":"2022-07-20T15:33:46.367Z","message":"Creating scope :manual. Overwriting existing method Ci::Stage.manual."}
{"severity":"WARN","time":"2022-07-20T15:33:46.367Z","message":"Creating scope :scheduled. Overwriting existing method Ci::Stage.scheduled."}
{"severity":"WARN","time":"2022-07-20T15:33:46.368Z","message":"Creating scope :preparing. Overwriting existing method Ci::Stage.preparing."}
{"severity":"WARN","time":"2022-07-20T15:33:46.368Z","message":"Creating scope :waiting_for_resource. Overwriting existing method Ci::Stage.waiting_for_resource."}
{"severity":"WARN","time":"2022-07-20T15:33:46.406Z","message":"Creating scope :database. Overwriting existing method Ci::BuildTraceChunk.database."}
{"severity":"WARN","time":"2022-07-20T15:33:48.213Z","message":"Creating scope :order_created_desc. Overwriting existing method RequirementsManagement::Requirement.order_created_desc."}
{"severity":"WARN","time":"2022-07-20T15:33:48.214Z","message":"Creating scope :order_created_asc. Overwriting existing method RequirementsManagement::Requirement.order_created_asc."}
{"severity":"WARN","time":"2022-07-20T15:33:48.214Z","message":"Creating scope :order_updated_desc. Overwriting existing method RequirementsManagement::Requirement.order_updated_desc."}
{"severity":"WARN","time":"2022-07-20T15:33:48.214Z","message":"Creating scope :order_updated_asc. Overwriting existing method RequirementsManagement::Requirement.order_updated_asc."}
{"severity":"WARN","time":"2022-07-20T15:33:48.236Z","message":"Creating scope :runnable_schedules. Overwriting existing method Security::OrchestrationPolicyRuleSchedule.runnable_schedules."}
{"severity":"WARN","time":"2022-07-20T15:33:49.886Z","message":"Creating scope :regular. Overwriting existing method EE::Gitlab::BackgroundMigration::MigrateApproverToApprovalRules::ApprovalMergeRequestRule.regular."}
{"severity":"WARN","time":"2022-07-20T15:33:49.887Z","message":"Creating scope :code_owner. Overwriting existing method EE::Gitlab::BackgroundMigration::MigrateApproverToApprovalRules::ApprovalMergeRequestRule.code_owner."}
{"severity":"WARN","time":"2022-07-20T15:33:54.206Z","message":"Creating scope :system. Overwriting existing method Gitlab::DatabaseImporters::CommonMetrics::PrometheusMetric.system."}
{"severity":"INFO","time":"2022-07-20T15:34:15.958Z","message":"Booted Rails 6.1.4.7 application in production environment"}
{"severity":"INFO","time":"2022-07-20T15:34:15.959Z","message":"Running in ruby 2.7.5p203 (2021-11-24 revision f69aeb8314) [x86_64-linux]"}
{"severity":"INFO","time":"2022-07-20T15:34:15.959Z","message":"See LICENSE and the LGPL-3.0 for licensing details."}
{"severity":"INFO","time":"2022-07-20T15:34:15.959Z","message":"Upgrade to Sidekiq Pro for more features and support: https://sidekiq.org"}
[2022-07-20 16:34:15] INFO  WEBrick 1.6.1
[2022-07-20 16:34:15] INFO  ruby 2.7.5 (2021-11-24) [x86_64-linux]
{"severity":"INFO","time":"2022-07-20T15:34:18.722Z","message":"A worker terminated, shutting down the cluster"}
{"severity":"INFO","time":"2022-07-20T15:34:25.318Z","message":"Starting cluster with 1 processes"}
{"severity":"INFO","time":"2022-07-20T15:34:46.734Z","message":"Listening on queues [\"adjourned_project_deletion\", \"a

I added more of the log in case you know what those warnings are
also when I check the background tasks form the command line :

sudo gitlab-rails runner -e production 'puts Gitlab::BackgroundMigration.remaining'

then it shows 4 but then the rest of the doc is not working

StephaneK · July 20, 2022, 3:44pm

oups lost my previous reply, so the server should have enough cpu/ram there is 24cores/64gb of ram
only half is used
concerning the log I remove the memorykiller with env variable
but the sidekiq log seems to still show constant reboot

{"severity":"INFO","time":"2022-07-20T15:42:07.135Z","message":"Cron Jobs - add job with name: geo_sidekiq_cron_config_worker"}
{"severity":"WARN","time":"2022-07-20T15:42:09.188Z","message":"Creating scope :without_statuses. Overwriting existing method Ci::Pipeline.without_statuses."}
{"severity":"WARN","time":"2022-07-20T15:42:09.996Z","message":"Creating scope :of_projects. Overwriting existing method MergeRequest.of_projects."}
{"severity":"WARN","time":"2022-07-20T15:42:09.998Z","message":"Creating scope :join_project. Overwriting existing method MergeRequest.join_project."}
{"severity":"WARN","time":"2022-07-20T15:42:09.998Z","message":"Creating scope :references_project. Overwriting existing method MergeRequest.references_project."}
{"severity":"WARN","time":"2022-07-20T15:42:10.063Z","message":"Creating scope :opened. Overwriting existing method Epic.opened."}
{"severity":"WARN","time":"2022-07-20T15:42:10.063Z","message":"Creating scope :closed. Overwriting existing method Epic.closed."}
{"severity":"WARN","time":"2022-07-20T15:42:10.080Z","message":"Creating scope :with_api_entity_associations. Overwriting existing method Epic.with_api_entity_associations."}
{"severity":"WARN","time":"2022-07-20T15:42:10.318Z","message":"Creating scope :system. Overwriting existing method Note.system."}
{"severity":"WARN","time":"2022-07-20T15:42:10.597Z","message":"Creating scope :order_created_desc. Overwriting existing method Packages::Package.order_created_desc."}
{"severity":"WARN","time":"2022-07-20T15:42:10.597Z","message":"Creating scope :order_name_desc. Overwriting existing method Packages::Package.order_name_desc."}
{"severity":"WARN","time":"2022-07-20T15:42:10.950Z","message":"Creating scope :closed. Overwriting existing method Iteration.closed."}
{"severity":"WARN","time":"2022-07-20T15:42:10.951Z","message":"Creating scope :within_timeframe. Overwriting existing method Iteration.within_timeframe."}
{"severity":"WARN","time":"2022-07-20T15:42:13.443Z","message":"Creating scope :open. Overwriting existing method AlertManagement::Alert.open."}
{"severity":"WARN","time":"2022-07-20T15:42:13.492Z","message":"Creating scope :open. Overwriting existing method IncidentManagement::IssuableEscalationStatus.open."}
{"severity":"WARN","time":"2022-07-20T15:42:13.599Z","message":"Creating scope :with_associations. Overwriting existing method Board.with_associations."}
{"severity":"WARN","time":"2022-07-20T15:42:13.717Z","message":"Creating scope :movable. Overwriting existing method Boards::EpicList.movable."}
{"severity":"WARN","time":"2022-07-20T15:42:14.434Z","message":"Creating scope :order_name_asc. Overwriting existing method Label.order_name_asc."}
{"severity":"WARN","time":"2022-07-20T15:42:14.434Z","message":"Creating scope :order_name_desc. Overwriting existing method Label.order_name_desc."}
{"severity":"WARN","time":"2022-07-20T15:42:15.270Z","message":"Creating scope :open. Overwriting existing method ExternalPullRequest.open."}
{"severity":"WARN","time":"2022-07-20T15:42:15.418Z","message":"Creating scope :order_name_asc. Overwriting existing method Member.order_name_asc."}
{"severity":"WARN","time":"2022-07-20T15:42:15.419Z","message":"Creating scope :order_name_desc. Overwriting existing method Member.order_name_desc."}
{"severity":"WARN","time":"2022-07-20T15:42:15.541Z","message":"Creating scope :with_logging_info. Overwriting existing method PagesDomain.with_logging_info."}
{"severity":"WARN","time":"2022-07-20T15:42:15.732Z","message":"Creating scope :system. Overwriting existing method PrometheusMetric.system."}
{"severity":"WARN","time":"2022-07-20T15:42:16.105Z","message":"Creating scope :with_user. Overwriting existing method GroupMember.with_user."}
{"severity":"WARN","time":"2022-07-20T15:42:16.109Z","message":"Creating scope :reporters. Overwriting existing method GroupMember.reporters."}
{"severity":"WARN","time":"2022-07-20T15:42:16.109Z","message":"Creating scope :guests. Overwriting existing method GroupMember.guests."}
{"severity":"WARN","time":"2022-07-20T15:42:16.448Z","message":"Creating scope :extended. Overwriting existing method GitlabSubscription.extended."}
{"severity":"WARN","time":"2022-07-20T15:42:17.105Z","message":"Creating scope :without_statuses. Overwriting existing method Ci::Stage.without_statuses."}
{"severity":"WARN","time":"2022-07-20T15:42:17.106Z","message":"Creating scope :created. Overwriting existing method Ci::Stage.created."}
{"severity":"WARN","time":"2022-07-20T15:42:17.106Z","message":"Creating scope :pending. Overwriting existing method Ci::Stage.pending."}
{"severity":"WARN","time":"2022-07-20T15:42:17.106Z","message":"Creating scope :running. Overwriting existing method Ci::Stage.running."}
{"severity":"WARN","time":"2022-07-20T15:42:17.107Z","message":"Creating scope :success. Overwriting existing method Ci::Stage.success."}
{"severity":"WARN","time":"2022-07-20T15:42:17.107Z","message":"Creating scope :failed. Overwriting existing method Ci::Stage.failed."}
{"severity":"WARN","time":"2022-07-20T15:42:17.107Z","message":"Creating scope :canceled. Overwriting existing method Ci::Stage.canceled."}
{"severity":"WARN","time":"2022-07-20T15:42:17.108Z","message":"Creating scope :skipped. Overwriting existing method Ci::Stage.skipped."}
{"severity":"WARN","time":"2022-07-20T15:42:17.108Z","message":"Creating scope :manual. Overwriting existing method Ci::Stage.manual."}
{"severity":"WARN","time":"2022-07-20T15:42:17.108Z","message":"Creating scope :scheduled. Overwriting existing method Ci::Stage.scheduled."}
{"severity":"WARN","time":"2022-07-20T15:42:17.109Z","message":"Creating scope :preparing. Overwriting existing method Ci::Stage.preparing."}
{"severity":"WARN","time":"2022-07-20T15:42:17.109Z","message":"Creating scope :waiting_for_resource. Overwriting existing method Ci::Stage.waiting_for_resource."}
{"severity":"WARN","time":"2022-07-20T15:42:17.137Z","message":"Creating scope :database. Overwriting existing method Ci::BuildTraceChunk.database."}
{"severity":"WARN","time":"2022-07-20T15:42:18.447Z","message":"Creating scope :order_created_desc. Overwriting existing method RequirementsManagement::Requirement.order_created_desc."}
{"severity":"WARN","time":"2022-07-20T15:42:18.447Z","message":"Creating scope :order_created_asc. Overwriting existing method RequirementsManagement::Requirement.order_created_asc."}
{"severity":"WARN","time":"2022-07-20T15:42:18.448Z","message":"Creating scope :order_updated_desc. Overwriting existing method RequirementsManagement::Requirement.order_updated_desc."}
{"severity":"WARN","time":"2022-07-20T15:42:18.448Z","message":"Creating scope :order_updated_asc. Overwriting existing method RequirementsManagement::Requirement.order_updated_asc."}
{"severity":"WARN","time":"2022-07-20T15:42:18.466Z","message":"Creating scope :runnable_schedules. Overwriting existing method Security::OrchestrationPolicyRuleSchedule.runnable_schedules."}
{"severity":"WARN","time":"2022-07-20T15:42:19.711Z","message":"Creating scope :regular. Overwriting existing method EE::Gitlab::BackgroundMigration::MigrateApproverToApprovalRules::ApprovalMergeRequestRule.regular."}
{"severity":"WARN","time":"2022-07-20T15:42:19.712Z","message":"Creating scope :code_owner. Overwriting existing method EE::Gitlab::BackgroundMigration::MigrateApproverToApprovalRules::ApprovalMergeRequestRule.code_owner."}
{"severity":"WARN","time":"2022-07-20T15:42:23.117Z","message":"Creating scope :system. Overwriting existing method Gitlab::DatabaseImporters::CommonMetrics::PrometheusMetric.system."}
{"severity":"INFO","time":"2022-07-20T15:42:41.863Z","message":"Booted Rails 6.1.4.7 application in production environment"}
{"severity":"INFO","time":"2022-07-20T15:42:41.863Z","message":"Running in ruby 2.7.5p203 (2021-11-24 revision f69aeb8314) [x86_64-linux]"}
{"severity":"INFO","time":"2022-07-20T15:42:41.863Z","message":"See LICENSE and the LGPL-3.0 for licensing details."}
{"severity":"INFO","time":"2022-07-20T15:42:41.863Z","message":"Upgrade to Sidekiq Pro for more features and support: https://sidekiq.org"}
[2022-07-20 16:42:41] INFO  WEBrick 1.6.1
[2022-07-20 16:42:41] INFO  ruby 2.7.5 (2021-11-24) [x86_64-linux]
{"severity":"INFO","time":"2022-07-20T15:42:45.112Z","message":"A worker terminated, shutting down the cluster"}
{"severity":"INFO","time":"2022-07-20T15:42:51.587Z","message":"Starting cluster with 1 processes"}

I included more of the sidekiq log in case you know what the warnings are
I will be able to fully reboot the server after work hours but I found it suspicious only sidekiq would be having problem

iwalker · July 20, 2022, 3:46pm

Can you run htop or top and find out what is hogging all the cpu. I’m curious what 13.x version you were on as there was a critical CVE about servers being compromised and CPU miners running on the system. Although that said, if your server is not publicly available on the internet then this shouldn’t be the problem.

You may want to check your config vs this: Sidekiq MemoryKiller | GitLab

StephaneK · July 20, 2022, 3:56pm

resources on the server seems fine,

also the server is not reachable from the internet, so would be unlikely that it could get corrupted from CPU miner, the previous version I was on was 13.12.15

iwalker · July 20, 2022, 4:16pm

Yeah just wanted to rule that out. Double-check to see if any of the gitlab config was changed in relation to the sidekiq memorykiller or whether it is the default - maybe something shows up in /etc/gitlab/gitlab.rb.

But, if you’ve removed all the ruby stuff, and also restarted the server, you can check/test to see if your gitlab-rails commands now work or not:

gitlab-rake gitlab:check

since now there shouldn’t be the ruby conflicts. And then check/test to see if you can now make commits/merges, etc.

StephaneK · July 20, 2022, 4:48pm

I got to restart the server, took a while but after reboot it still shows the same error.
not sure what else to do now, maybe trying to rollback to the old version

Topic		Replies	Views
Sidekiq not running Self-managed	5	1908	August 2, 2024
Sidekiq won't start due to write error to sidekiq_exporter.log Self-managed	2	518	July 14, 2023
GitLab 7.5 -> 7.83 (Latest Version) Update Error How to Use GitLab	0	1086	March 12, 2015
Accept merge request freezing after update 8.14 [ SOLVED ] Self-managed	1	825	November 28, 2016
Unable to merge How to Use GitLab	5	2293	June 27, 2017

Cant merge after upgrade (sidekiq related ?)

Related topics