Sidekiq Cluster keeps terminating

Hi!!

I installed GitLab CE from source, and everything is working perfectly. That is, everything except Sidekiq.

It runs for ~2 seconds, and then just terminates without giving any reason why.

Below is the log:

{"severity":"INFO","time":"2021-11-01T21:51:11.429Z","message":"A worker terminated, shutting down the cluster"}
{"severity":"INFO","time":"2021-11-01T22:36:37.700Z","message":"Starting cluster with 1 processes"}
2021-11-01T22:37:04.016Z pid=1126513 tid=ogqh INFO: Listening on queues ["analytics_usage_trends_counter_job", "approve_blocked_pending_approval_users", "authorized_keys", "authorized_project_update:authorized_project_update_project_create", "authorized_project_update:authorized_project_update_project_group_link_create", "authorized_project_update:authorized_project_update_project_recalculate", "authorized_project_update:authorized_project_update_project_recalculate_per_user", "authorized_project_update:authorized_project_update_user_refresh_from_replica", "authorized_project_update:authorized_project_update_user_refresh_over_user_range", "authorized_project_update:authorized_project_update_user_refresh_with_low_urgency", "authorized_projects", "auto_devops:auto_devops_disable", "auto_merge:auto_merge_process", "background_migration", "bulk_import", "bulk_imports_entity", "bulk_imports_export_request", "bulk_imports_pipeline", "bulk_imports_relation_export", "chaos:chaos_cpu_spin", "chaos:chaos_db_spin", "chaos:chaos_kill", "chaos:chaos_leak_mem", "chaos:chaos_sleep", "chat_notification", "ci_delete_objects", "container_repository:cleanup_container_repository", "container_repository:container_expiration_policies_cleanup_container_repository", "container_repository:delete_container_repository", "create_commit_signature", "create_note_diff_file", "cronjob:admin_email", "cronjob:analytics_usage_trends_count_job_trigger", "cronjob:authorized_project_update_periodic_recalculate", "cronjob:ci_archive_traces_cron", "cronjob:ci_delete_unit_tests", "cronjob:ci_pipeline_artifacts_expire_artifacts", "cronjob:ci_platform_metrics_update_cron", "cronjob:ci_schedule_delete_objects_cron", "cronjob:ci_stuck_builds_drop_running", "cronjob:ci_stuck_builds_drop_scheduled", "cronjob:container_expiration_policy", "cronjob:database_batched_background_migration", "cronjob:database_drop_detached_partitions", "cronjob:database_partition_management", "cronjob:dependency_proxy_image_ttl_group_policy", "cronjob:environments_auto_delete_cron", "cronjob:environments_auto_stop_cron", "cronjob:expire_build_artifacts", "cronjob:gitlab_service_ping", "cronjob:import_export_project_cleanup", "cronjob:import_stuck_project_import_jobs", "cronjob:issue_due_scheduler", "cronjob:jira_import_stuck_jira_import_jobs", "cronjob:member_invitation_reminder_emails", "cronjob:metrics_dashboard_schedule_annotations_prune", "cronjob:namespaces_in_product_marketing_emails", "cronjob:namespaces_prune_aggregation_schedules", "cronjob:packages_composer_cache_cleanup", "cronjob:pages_domain_removal_cron", "cronjob:pages_domain_ssl_renewal_cron", "cronjob:pages_domain_verification_cron", "cronjob:partition_creation", "cronjob:personal_access_tokens_expired_notification", "cronjob:personal_access_tokens_expiring", "cronjob:pipeline_schedule", "cronjob:prune_old_events", "cronjob:releases_manage_evidence", "cronjob:remove_expired_group_links", "cronjob:remove_expired_members", "cronjob:remove_unaccepted_member_invites", "cronjob:remove_unreferenced_lfs_objects", "cronjob:repository_archive_cache", "cronjob:repository_check_dispatch", "cronjob:requests_profiles", "cronjob:schedule_merge_request_cleanup_refs", "cronjob:schedule_migrate_external_diffs", "cronjob:ssh_keys_expired_notification", "cronjob:ssh_keys_expiring_soon_notification", "cronjob:stuck_ci_jobs", "cronjob:stuck_export_jobs", "cronjob:stuck_merge_jobs", "cronjob:trending_projects", "cronjob:update_container_registry_info", "cronjob:user_status_cleanup_batch", "cronjob:users_create_statistics", "cronjob:users_deactivate_dormant_users", "cronjob:x509_issuer_crl_check", "default", "delete_diff_files", "delete_merged_branches", "delete_stored_files", "delete_user", "dependency_proxy:purge_dependency_proxy_cache", "dependency_proxy_blob:dependency_proxy_cleanup_blob", "dependency_proxy_manifest:dependency_proxy_cleanup_manifest", "deployment:deployments_drop_older_deployments", "deployment:deployments_hooks", "deployment:deployments_link_merge_request", "deployment:deployments_update_environment", "design_management_copy_design_collection", "design_management_new_version", "destroy_pages_deployments", "detect_repository_languages", "disallow_two_factor_for_group", "disallow_two_factor_for_subgroups", "email_receiver", "emails_on_push", "environments_auto_stop", "environments_canary_ingress_update", "error_tracking_issue_link", "experiments_record_conversion_event", "expire_build_instance_artifacts", "export_csv", "external_service_reactive_caching", "file_hook", "flush_counter_increments", "gcp_cluster:cluster_configure_istio", "gcp_cluster:cluster_install_app", "gcp_cluster:cluster_patch_app", "gcp_cluster:cluster_provision", "gcp_cluster:cluster_update_app", "gcp_cluster:cluster_upgrade_app", "gcp_cluster:cluster_wait_for_app_installation", "gcp_cluster:cluster_wait_for_app_update", "gcp_cluster:cluster_wait_for_ingress_ip_address", "gcp_cluster:clusters_applications_activate_service", "gcp_cluster:clusters_applications_deactivate_service", "gcp_cluster:clusters_applications_uninstall", "gcp_cluster:clusters_applications_wait_for_uninstall_app", "gcp_cluster:clusters_cleanup_project_namespace", "gcp_cluster:clusters_cleanup_service_account", "gcp_cluster:wait_for_cluster_creation", "github_import_advance_stage", "github_importer:github_import_import_diff_note", "github_importer:github_import_import_issue", "github_importer:github_import_import_lfs_object", "github_importer:github_import_import_note", "github_importer:github_import_import_pull_request", "github_importer:github_import_import_pull_request_merged_by", "github_importer:github_import_import_pull_request_review", "github_importer:github_import_refresh_import_jid", "github_importer:github_import_stage_finish_import", "github_importer:github_import_stage_import_base_data", "github_importer:github_import_stage_import_issues_and_diff_notes", "github_importer:github_import_stage_import_lfs_objects", "github_importer:github_import_stage_import_notes", "github_importer:github_import_stage_import_pull_requests", "github_importer:github_import_stage_import_pull_requests_merged_by", "github_importer:github_import_stage_import_pull_requests_reviews", "github_importer:github_import_stage_import_repository", "gitlab_performance_bar_stats", "gitlab_shell", "group_destroy", "group_export", "group_import", "hashed_storage:hashed_storage_migrator", "hashed_storage:hashed_storage_project_migrate", "hashed_storage:hashed_storage_project_rollback", "hashed_storage:hashed_storage_rollbacker", "import_issues_csv", "incident_management:clusters_applications_check_prometheus_health", "incident_management:incident_management_add_severity_system_note", "incident_management:incident_management_pager_duty_process_incident", "incident_management:incident_management_process_alert_worker_v2", "invalid_gpg_signature_update", "irker", "issuable_export_csv", "issuable_label_links_destroy", "issuables_clear_groups_issue_counter", "issue_placement", "issue_rebalancing", "jira_connect:jira_connect_forward_event", "jira_connect:jira_connect_retry_request", "jira_connect:jira_connect_sync_branch", "jira_connect:jira_connect_sync_builds", "jira_connect:jira_connect_sync_deployments", "jira_connect:jira_connect_sync_feature_flags", "jira_connect:jira_connect_sync_merge_request", "jira_connect:jira_connect_sync_project", "jira_importer:jira_import_advance_stage", "jira_importer:jira_import_import_issue", "jira_importer:jira_import_stage_finish_import", "jira_importer:jira_import_stage_import_attachments", "jira_importer:jira_import_stage_import_issues", "jira_importer:jira_import_stage_import_labels", "jira_importer:jira_import_stage_import_notes", "jira_importer:jira_import_stage_start_import", "mail_scheduler:mail_scheduler_issue_due", "mail_scheduler:mail_scheduler_notification_service", "mailers", "merge", "merge_request_cleanup_refs", "merge_request_mergeability_check", "merge_requests_delete_source_branch", "merge_requests_handle_assignees_change", "merge_requests_resolve_todos", "metrics_dashboard_prune_old_annotations", "metrics_dashboard_sync_dashboards", "migrate_external_diffs", "namespaceless_project_destroy", "namespaces_onboarding_issue_created", "namespaces_onboarding_pipeline_created", "namespaces_onboarding_progress", "namespaces_onboarding_user_added", "new_issue", "new_merge_request", "new_note", "object_pool:object_pool_create", "object_pool:object_pool_destroy", "object_pool:object_pool_join", "object_pool:object_pool_schedule_join", "object_storage:object_storage_background_move", "object_storage:object_storage_migrate_uploads", "package_repositories:packages_debian_generate_distribution", "package_repositories:packages_debian_process_changes", "package_repositories:packages_go_sync_packages", "package_repositories:packages_helm_extraction", "package_repositories:packages_maven_metadata_sync", "package_repositories:packages_nuget_extraction", "package_repositories:packages_rubygems_extraction", "packages_composer_cache_update", "pages", "pages_domain_ssl_renewal", "pages_domain_verification", "pages_transfer", "pages_update_configuration", "phabricator_import_import_tasks", "pipeline_background:archive_trace", "pipeline_background:ci_archive_trace", "pipeline_background:ci_build_trace_chunk_flush", "pipeline_background:ci_daily_build_group_report_results", "pipeline_background:ci_pipeline_artifacts_coverage_report", "pipeline_background:ci_pipeline_artifacts_create_quality_report", "pipeline_background:ci_pipeline_success_unlock_artifacts", "pipeline_background:ci_ref_delete_unlock_artifacts", "pipeline_background:ci_test_failure_history", "pipeline_cache:expire_job_cache", "pipeline_cache:expire_pipeline_cache", "pipeline_creation:ci_external_pull_requests_create_pipeline", "pipeline_creation:create_pipeline", "pipeline_creation:merge_requests_create_pipeline", "pipeline_creation:run_pipeline_schedule", "pipeline_default:ci_create_cross_project_pipeline", "pipeline_default:ci_create_downstream_pipeline", "pipeline_default:ci_drop_pipeline", "pipeline_default:ci_merge_requests_add_todo_when_build_fails", "pipeline_default:ci_pipeline_bridge_status", "pipeline_default:ci_retry_pipeline", "pipeline_default:pipeline_metrics", "pipeline_default:pipeline_notification", "pipeline_hooks:build_hooks", "pipeline_hooks:pipeline_hooks", "pipeline_processing:build_finished", "pipeline_processing:build_queue", "pipeline_processing:build_success", "pipeline_processing:ci_build_finished", "pipeline_processing:ci_build_prepare", "pipeline_processing:ci_build_schedule", "pipeline_processing:ci_initial_pipeline_process", "pipeline_processing:ci_resource_groups_assign_resource_from_resource_group", "pipeline_processing:pipeline_process", "pipeline_processing:stage_update", "pipeline_processing:update_head_pipeline_for_merge_request", "post_receive", "process_commit", "project_cache", "project_daily_statistics", "project_destroy", "project_export", "project_service", "projects_git_garbage_collect", "projects_post_creation", "projects_schedule_bulk_repository_shard_moves", "projects_update_repository_storage", "prometheus_create_default_alerts", "propagate_integration", "propagate_integration_group", "propagate_integration_inherit", "propagate_integration_inherit_descendant", "propagate_integration_project", "propagate_service_template", "reactive_caching", "rebase", "releases_create_evidence", "remote_mirror_notification", "repository_check:repository_check_batch", "repository_check:repository_check_clear", "repository_check:repository_check_single_repository", "repository_cleanup", "repository_fork", "repository_import", "repository_remove_remote", "repository_update_remote_mirror", "self_monitoring_project_create", "self_monitoring_project_delete", "service_desk_email_receiver", "snippets_schedule_bulk_repository_shard_moves", "snippets_update_repository_storage", "system_hook_push", "todos_destroyer:todos_destroyer_confidential_issue", "todos_destroyer:todos_destroyer_destroyed_designs", "todos_destroyer:todos_destroyer_destroyed_issuable", "todos_destroyer:todos_destroyer_entity_leave", "todos_destroyer:todos_destroyer_group_private", "todos_destroyer:todos_destroyer_private_features", "todos_destroyer:todos_destroyer_project_private", "unassign_issuables:members_destroyer_unassign_issuables", "update_external_pull_requests", "update_highest_role", "update_merge_requests", "update_namespace_statistics:namespaces_root_statistics", "update_namespace_statistics:namespaces_schedule_aggregation", "update_project_statistics", "upload_checksum", "web_hook", "web_hooks_destroy", "web_hooks_log_execution", "wikis_git_garbage_collect", "x509_certificate_revoke"]
{"severity":"INFO","time":"2021-11-01T22:37:04.018Z","message":"Booting Sidekiq 6.2.2 with redis options {:instrumentation_class=\u003eGitlab::Instrumentation::Redis::Queues, :path=\u003e\"/var/run/redis/redis.sock\", :namespace=\u003e\"resque:gitlab\"}"}
{"severity":"INFO","time":"2021-11-01T22:37:05.143Z","message":"GitLab reliable fetch activated!"}
{"severity":"INFO","time":"2021-11-01T22:37:05.197Z","message":"Cron Jobs - add job with name: stuck_ci_jobs_worker"}
{"severity":"INFO","time":"2021-11-01T22:37:05.202Z","message":"Cron Jobs - add job with name: pipeline_schedule_worker"}
{"severity":"INFO","time":"2021-11-01T22:37:05.206Z","message":"Cron Jobs - add job with name: expire_build_artifacts_worker"}
{"severity":"INFO","time":"2021-11-01T22:37:05.209Z","message":"Cron Jobs - add job with name: ci_pipelines_expire_artifacts_worker"}
{"severity":"INFO","time":"2021-11-01T22:37:05.213Z","message":"Cron Jobs - add job with name: ci_schedule_delete_objects_worker"}
{"severity":"INFO","time":"2021-11-01T22:37:05.217Z","message":"Cron Jobs - add job with name: environments_auto_stop_cron_worker"}
{"severity":"INFO","time":"2021-11-01T22:37:05.221Z","message":"Cron Jobs - add job with name: environments_auto_delete_cron_worker"}
{"severity":"INFO","time":"2021-11-01T22:37:05.224Z","message":"Cron Jobs - add job with name: repository_check_worker"}
{"severity":"INFO","time":"2021-11-01T22:37:05.228Z","message":"Cron Jobs - add job with name: ci_archive_traces_cron_worker"}
{"severity":"INFO","time":"2021-11-01T22:37:05.232Z","message":"Cron Jobs - add job with name: admin_email_worker"}
{"severity":"INFO","time":"2021-11-01T22:37:05.236Z","message":"Cron Jobs - add job with name: personal_access_tokens_expiring_worker"}
{"severity":"INFO","time":"2021-11-01T22:37:05.240Z","message":"Cron Jobs - add job with name: repository_archive_cache_worker"}
{"severity":"INFO","time":"2021-11-01T22:37:05.244Z","message":"Cron Jobs - add job with name: pages_domain_verification_cron_worker"}
{"severity":"INFO","time":"2021-11-01T22:37:05.247Z","message":"Cron Jobs - add job with name: schedule_migrate_external_diffs_worker"}
{"severity":"INFO","time":"2021-11-01T22:37:05.251Z","message":"Cron Jobs - add job with name: ci_platform_metrics_update_cron_worker"}
{"severity":"INFO","time":"2021-11-01T22:37:05.255Z","message":"Cron Jobs - add job with name: personal_access_tokens_expired_notification_worker"}
{"severity":"INFO","time":"2021-11-01T22:37:05.258Z","message":"Cron Jobs - add job with name: import_export_project_cleanup_worker"}
{"severity":"INFO","time":"2021-11-01T22:37:05.262Z","message":"Cron Jobs - add job with name: requests_profiles_worker"}
{"severity":"INFO","time":"2021-11-01T22:37:05.265Z","message":"Cron Jobs - add job with name: remove_expired_members_worker"}
{"severity":"INFO","time":"2021-11-01T22:37:05.269Z","message":"Cron Jobs - add job with name: remove_expired_group_links_worker"}
{"severity":"INFO","time":"2021-11-01T22:37:05.272Z","message":"Cron Jobs - add job with name: remove_unaccepted_member_invites_worker"}
{"severity":"INFO","time":"2021-11-01T22:37:05.276Z","message":"Cron Jobs - add job with name: prune_old_events_worker"}
{"severity":"INFO","time":"2021-11-01T22:37:05.280Z","message":"Cron Jobs - add job with name: trending_projects_worker"}
{"severity":"INFO","time":"2021-11-01T22:37:05.283Z","message":"Cron Jobs - add job with name: remove_unreferenced_lfs_objects_worker"}
{"severity":"INFO","time":"2021-11-01T22:37:05.287Z","message":"Cron Jobs - add job with name: import_stuck_project_import_jobs"}
{"severity":"INFO","time":"2021-11-01T22:37:05.290Z","message":"Cron Jobs - add job with name: jira_import_stuck_jira_import_jobs"}
{"severity":"INFO","time":"2021-11-01T22:37:05.293Z","message":"Cron Jobs - add job with name: stuck_export_jobs_worker"}
{"severity":"INFO","time":"2021-11-01T22:37:05.297Z","message":"Cron Jobs - add job with name: gitlab_service_ping_worker"}
{"severity":"INFO","time":"2021-11-01T22:37:05.301Z","message":"Cron Jobs - add job with name: stuck_merge_jobs_worker"}
{"severity":"INFO","time":"2021-11-01T22:37:05.306Z","message":"Cron Jobs - add job with name: pages_domain_removal_cron_worker"}
{"severity":"INFO","time":"2021-11-01T22:37:05.312Z","message":"Cron Jobs - add job with name: pages_domain_ssl_renewal_cron_worker"}
{"severity":"INFO","time":"2021-11-01T22:37:05.317Z","message":"Cron Jobs - add job with name: issue_due_scheduler_worker"}
{"severity":"INFO","time":"2021-11-01T22:37:05.322Z","message":"Cron Jobs - add job with name: metrics_dashboard_schedule_annotations_prune_worker"}
{"severity":"INFO","time":"2021-11-01T22:37:05.328Z","message":"Cron Jobs - add job with name: namespaces_prune_aggregation_schedules_worker"}
{"severity":"INFO","time":"2021-11-01T22:37:05.544Z","message":"Cron Jobs - add job with name: container_expiration_policy_worker"}
{"severity":"INFO","time":"2021-11-01T22:37:05.551Z","message":"Cron Jobs - add job with name: image_ttl_group_policy_worker"}
{"severity":"INFO","time":"2021-11-01T22:37:05.554Z","message":"Cron Jobs - add job with name: x509_issuer_crl_check_worker"}
{"severity":"INFO","time":"2021-11-01T22:37:05.556Z","message":"Cron Jobs - add job with name: users_create_statistics_worker"}
{"severity":"INFO","time":"2021-11-01T22:37:05.558Z","message":"Cron Jobs - add job with name: authorized_project_update_periodic_recalculate_worker"}
{"severity":"INFO","time":"2021-11-01T22:37:05.561Z","message":"Cron Jobs - add job with name: update_container_registry_info_worker"}
{"severity":"INFO","time":"2021-11-01T22:37:05.563Z","message":"Cron Jobs - add job with name: postgres_dynamic_partitions_manager"}
{"severity":"INFO","time":"2021-11-01T22:37:05.565Z","message":"Cron Jobs - add job with name: postgres_dynamic_partitions_dropper"}
{"severity":"INFO","time":"2021-11-01T22:37:05.567Z","message":"Cron Jobs - add job with name: analytics_usage_trends_count_job_trigger_worker"}
{"severity":"INFO","time":"2021-11-01T22:37:05.569Z","message":"Cron Jobs - add job with name: member_invitation_reminder_emails_worker"}
{"severity":"INFO","time":"2021-11-01T22:37:05.571Z","message":"Cron Jobs - add job with name: schedule_merge_request_cleanup_refs_worker"}
{"severity":"INFO","time":"2021-11-01T22:37:05.573Z","message":"Cron Jobs - add job with name: manage_evidence_worker"}
{"severity":"INFO","time":"2021-11-01T22:37:05.576Z","message":"Cron Jobs - add job with name: user_status_cleanup_batch_worker"}
{"severity":"INFO","time":"2021-11-01T22:37:05.578Z","message":"Cron Jobs - add job with name: ssh_keys_expired_notification_worker"}
{"severity":"INFO","time":"2021-11-01T22:37:05.580Z","message":"Cron Jobs - add job with name: namespaces_in_product_marketing_emails_worker"}
{"severity":"INFO","time":"2021-11-01T22:37:05.582Z","message":"Cron Jobs - add job with name: ssh_keys_expiring_soon_notification_worker"}
{"severity":"INFO","time":"2021-11-01T22:37:05.585Z","message":"Cron Jobs - add job with name: users_deactivate_dormant_users_worker"}
{"severity":"INFO","time":"2021-11-01T22:37:05.587Z","message":"Cron Jobs - add job with name: ci_delete_unit_tests_worker"}
{"severity":"INFO","time":"2021-11-01T22:37:05.590Z","message":"Cron Jobs - add job with name: batched_background_migrations_worker"}
{"severity":"INFO","time":"2021-11-01T22:37:31.091Z","message":"Booted Rails 6.1.4.1 application in production environment"}
{"severity":"INFO","time":"2021-11-01T22:37:31.091Z","message":"Running in ruby 2.7.4p191 (2021-07-07 revision a21a3b7d23) [x86_64-linux]"}
{"severity":"INFO","time":"2021-11-01T22:37:31.091Z","message":"See LICENSE and the LGPL-3.0 for licensing details."}
{"severity":"INFO","time":"2021-11-01T22:37:31.091Z","message":"Upgrade to Sidekiq Pro for more features and support: https://sidekiq.org"}
{"severity":"INFO","time":"2021-11-01T22:37:32.743Z","message":"A worker terminated, shutting down the cluster"}

I would appreciate some help, as I can’t do much without Sidekiq.

All I can say is that I have the same problem after upgrading to 14.6.0 (which is causing all kinds of weird issues because scheduled jobs do not get executed).

Have you been able to debug this? I tried to play around with Sidekiq queues in gitlab-rails console but that did not help much.

I also experience weird issues after upgrading to 14.6.0: Sidekiq queues jobs after 14.6.0 update

Sidekiq is not able to process some jobs, they are stuck in queue status. The log is exactly like yours, the warnings keeps repeating. How did you exactly attach a debugger to the sidekiq worker?

I’m not sure if it is relevant, the port 8082 is used by puma.