GitLab Sidekiq Broadcasting Unwanted Data to Google Cloud: How to Disable and Reduce Log Output

I’m encountering an issue with GitLab Sidekiq that appears to be sending data to Google Cloud (35.190.167.255) on every pageload, even though I have not configured it to do so. Despite my efforts to disable this behavior, I’ve been unsuccessful in resolving the problem. I am running the latest omnibus version.

Here’s what I’ve tried so far:

  1. Setting metrics_enabled to false in the configuration:
sidekiq['metrics_enabled'] = false
  1. Adjusting logging levels in various GitLab components:
gitaly['configuration'] = {
  logging_level: 'ERROR',
}

gitlab_rails['env'] = {
  "GITLAB_LOG_LEVEL" => "ERROR"
}

registry['log_level'] = 'error'
gitlab_shell['log_level'] = 'ERROR'

Despite these configurations, the issue persists, and GitLab Sidekiq continues to transmit data to Google Cloud on every pageload and job.

Could someone please assist me in disabling GitLab Sidekiq broadcasts to Google Cloud and also help me reduce the log output? Any insights and suggestions would be greatly appreciated.

I’ve already explored the following resources without success:

Thank you in advance for your assistance!

1 Like

It seems I have same problem. I hit into this exception:

{"severity":"ERROR"
"time":"2023-10-25T22:14:34.802Z"
"correlation_id":"0adb256e65b8d12de92a8d2e11b1310b"
"meta.caller_id":"ExternalServiceReactiveCachingWorker"
"meta.remote_ip":"172.30.50.3"
"meta.feature_category":"
"
"meta.user":"kami"
"meta.user_id":115
"meta.project":"hu.teszt.okr/xujb"
"meta.root_namespace":"hu.teszt.okr"
"meta.client_id":"user/115"
"meta.related_class":"VersionCheck"
"meta.root_caller_id":"Projects::MergeRequestsController#index"
"exception.class":"Errno::ECONNREFUSED"
"exception.message":"Failed to open TCP connection to 35.190.167.255:443 (Connection refused - connect(2) for \"35.190.167.255\" port 443)"
"exception.backtrace":["lib/gitlab/net_http_adapter.rb:21:in `connect'"
"lib/gitlab/http.rb:69:in `perform_request'"
"lib/gitlab/http.rb:89:in `try_get'"
"lib/version_check.rb:71:in `calculate_reactive_cache'"
"app/models/concerns/reactive_caching.rb:94:in `block (2 levels) in exclusively_update_reactive_cache!'"
"app/models/concerns/reactive_caching.rb:153:in `enqueuing_update'"
"app/models/concerns/reactive_caching.rb:93:in `block in exclusively_update_reactive_cache!'"
"app/models/concerns/reactive_caching.rb:143:in `locking_reactive_cache'"
"app/models/concerns/reactive_caching.rb:89:in `exclusively_update_reactive_cache!'"
"app/workers/concerns/reactive_cacheable_worker.rb:35:in `perform'"
"lib/gitlab/sidekiq_middleware/skip_jobs.rb:49:in `call'"
"lib/gitlab/database/load_balancing/sidekiq_server_middleware.rb:29:in `call'"
"lib/gitlab/sidekiq_middleware/duplicate_jobs/strategies/until_executing.rb:16:in `perform'"
"lib/gitlab/sidekiq_middleware/duplicate_jobs/duplicate_job.rb:44:in `perform'"
"lib/gitlab/sidekiq_middleware/duplicate_jobs/server.rb:8:in `call'"
"lib/gitlab/sidekiq_middleware/pause_control/strategies/base.rb:31:in `perform'"
"lib/gitlab/sidekiq_middleware/pause_control/strategy_handler.rb:22:in `perform'"
"lib/gitlab/sidekiq_middleware/pause_control/server.rb:8:in `call'"
"lib/gitlab/sidekiq_middleware/worker_context.rb:9:in `wrap_in_optional_context'"
"lib/gitlab/sidekiq_middleware/worker_context/server.rb:19:in `block in call'"
"lib/gitlab/application_context.rb:124:in `block in use'"
"lib/gitlab/application_context.rb:124:in `use'"
"lib/gitlab/application_context.rb:62:in `with_context'"
"lib/gitlab/sidekiq_middleware/worker_context/server.rb:17:in `call'"
"lib/gitlab/sidekiq_status/server_middleware.rb:7:in `call'"
"lib/gitlab/sidekiq_versioning/middleware.rb:9:in `call'"
"lib/gitlab/sidekiq_middleware/query_analyzer.rb:7:in `block in call'"
"lib/gitlab/database/query_analyzer.rb:37:in `within'"
"lib/gitlab/sidekiq_middleware/query_analyzer.rb:7:in `call'"
"lib/gitlab/sidekiq_middleware/admin_mode/server.rb:14:in `call'"
"lib/gitlab/sidekiq_middleware/instrumentation_logger.rb:9:in `call'"
"lib/gitlab/sidekiq_middleware/batch_loader.rb:7:in `call'"
"lib/gitlab/sidekiq_middleware/extra_done_log_metadata.rb:7:in `call'"
"lib/gitlab/sidekiq_middleware/request_store_middleware.rb:8:in `block in call'"
"lib/gitlab/sidekiq_middleware/request_store_middleware.rb:7:in `call'"
"lib/gitlab/sidekiq_middleware/server_metrics.rb:94:in `block in call'"
"lib/gitlab/sidekiq_middleware/server_metrics.rb:122:in `block in instrument'"
"lib/gitlab/metrics/background_transaction.rb:33:in `run'"
"lib/gitlab/sidekiq_middleware/server_metrics.rb:122:in `instrument'"
"lib/gitlab/sidekiq_middleware/server_metrics.rb:93:in `call'"
"lib/gitlab/sidekiq_middleware/monitor.rb:10:in `block in call'"
"lib/gitlab/sidekiq_daemon/monitor.rb:46:in `within_job'"
"lib/gitlab/sidekiq_middleware/monitor.rb:9:in `call'"
"lib/gitlab/sidekiq_middleware/size_limiter/server.rb:13:in `call'"
"lib/gitlab/sidekiq_logging/structured_logger.rb:21:in `call'"]
"exception.cause_class":"Errno::ECONNREFUSED"
"user.username":"kami"
"tags.program":"sidekiq"
"tags.locale":"en"
"tags.feature_category":"code_review_workflow"
"tags.correlation_id":"0adb256e65b8d12de92a8d2e11b1310b"
"extra.sidekiq":{"retry":3
"queue":"default"
"version":0
"class":"ExternalServiceReactiveCachingWorker"
"args":["VersionCheck"
"[FILTERED]"]
"jid":"31e34d2f20504193525f03e1"
"created_at":1698228869.0104487
"correlation_id":"0adb256e65b8d12de92a8d2e11b1310b"
"meta.caller_id":"ExternalServiceReactiveCachingWorker"
"meta.remote_ip":"172.30.50.7"
"meta.feature_category":"code_review_workflow"
"meta.user":"kami"
"meta.user_id":115
"meta.project":"hu.teszt.okr/xujb"
"meta.root_namespace":"hu.teszt.okr"
"meta.client_id":"user/115"
"meta.related_class":"VersionCheck"
"meta.root_caller_id":"Projects::MergeRequestsController#index"
"worker_data_consistency":"always"
"size_limiter":"validated"
"scheduled_at":1698272069.0103996
"idempotency_key":"resque:gitlab:duplicate:default:87f4bdae2448a0477518235154a90b9ea4cc11406322e86e1123f5228f01c91e"
"enqueued_at":1698272074.7555292}}
1 Like