Since the 14.6.0 update I encounter a weird issue with the GitLab emails. No longer will any email be sent from GitLab to any account, but after I reboot the server then all emails are being sent together! Before the 14.6.0 update everything worked fine. I have not changed anything on the configuration and this issue exists also with the latest version (14.6.2). I use an omnibus installation on an Ubuntu 20.04 Server.
Has anybody experienced this?
I have latest version and email still works for all 14.6 versions.
Check your log files and verify your email configuration.
I use sendmail and from the rails console I can send an email without any problem. Moreover all the emails are sent when the server is restarted. It is like something blocks the emails from being sent when they are supposed to be, but after the restart all of them are sent bulky.
I have already checked the logs, but I will double checked them.
Check your mail logs when it isnt sending and make sure sendmail is running properly. You can use:
sendmail -q
To check the mail queue. Maybe the service is dying and restarting sendmail is enough. I think this is what is happening since you reboot and then it works which means sendmail would have started.
The mail queue is empty. I figured out that the problem is with sidekiq. It queues not only the emails, but much more (issues_placement, new_issue, projects_post_creation, projects_process_sync_events etc). According to the sidekiq log it seems that it continuously reboots. I am not sure how to interpret it.
I attach the part of the log that is repeated almost every 15 sec.
2022-01-15T11:38:07.896Z pid=9358 tid=num INFO: Listening on queues [“analytics_usage_trends_counter_job”, “approve_blocked_pending_approval_users”, “authorized_keys”, “authorized_project_update:authorized_project_update_project_create”, “authorized_project_update:authorized_project_update_project_group_link_create”, “authorized_project_update:authorized_project_update_project_recalculate”, “authorized_project_update:authorized_project_update_project_recalculate_per_user”, “authorized_project_update:authorized_project_update_user_refresh_from_replica”, “authorized_project_update:authorized_project_update_user_refresh_over_user_range”, “authorized_project_update:authorized_project_update_user_refresh_with_low_urgency”, “authorized_projects”, “auto_devops:auto_devops_disable”, “auto_merge:auto_merge_process”, “background_migration”, “bulk_import”, “bulk_imports_entity”, “bulk_imports_export_request”, “bulk_imports_pipeline”, “bulk_imports_relation_export”, “chaos:chaos_cpu_spin”, “chaos:chaos_db_spin”, “chaos:chaos_kill”, “chaos:chaos_leak_mem”, “chaos:chaos_sleep”, “chat_notification”, “ci_delete_objects”, “container_repository:cleanup_container_repository”, “container_repository:container_expiration_policies_cleanup_container_repository”, “container_repository:delete_container_repository”, “create_commit_signature”, “create_note_diff_file”, “cronjob:admin_email”, “cronjob:analytics_usage_trends_count_job_trigger”, “cronjob:authorized_project_update_periodic_recalculate”, “cronjob:ci_archive_traces_cron”, “cronjob:ci_delete_unit_tests”, “cronjob:ci_pipeline_artifacts_expire_artifacts”, “cronjob:ci_platform_metrics_update_cron”, “cronjob:ci_schedule_delete_objects_cron”, “cronjob:ci_stuck_builds_drop_running”, “cronjob:ci_stuck_builds_drop_scheduled”, “cronjob:clusters_integrations_check_prometheus_health”, “cronjob:container_expiration_policy”, “cronjob:database_batched_background_migration”, “cronjob:database_drop_detached_partitions”, “cronjob:database_partition_management”, “cronjob:dependency_proxy_cleanup_dependency_proxy”, “cronjob:dependency_proxy_image_ttl_group_policy”, “cronjob:environments_auto_delete_cron”, “cronjob:environments_auto_stop_cron”, “cronjob:expire_build_artifacts”, “cronjob:gitlab_service_ping”, “cronjob:import_export_project_cleanup”, “cronjob:import_stuck_project_import_jobs”, “cronjob:issue_due_scheduler”, “cronjob:issues_reschedule_stuck_issue_rebalances”, “cronjob:jira_import_stuck_jira_import_jobs”, “cronjob:loose_foreign_keys_cleanup”, “cronjob:member_invitation_reminder_emails”, “cronjob:metrics_dashboard_schedule_annotations_prune”, “cronjob:namespaces_in_product_marketing_emails”, “cronjob:namespaces_prune_aggregation_schedules”, “cronjob:packages_composer_cache_cleanup”, “cronjob:pages_domain_removal_cron”, “cronjob:pages_domain_ssl_renewal_cron”, “cronjob:pages_domain_verification_cron”, “cronjob:partition_creation”, “cronjob:personal_access_tokens_expired_notification”, “cronjob:personal_access_tokens_expiring”, “cronjob:pipeline_schedule”, “cronjob:prune_old_events”, “cronjob:releases_manage_evidence”, “cronjob:remove_expired_group_links”, “cronjob:remove_expired_members”, “cronjob:remove_unaccepted_member_invites”, “cronjob:remove_unreferenced_lfs_objects”, “cronjob:repository_archive_cache”, “cronjob:repository_check_dispatch”, “cronjob:requests_profiles”, “cronjob:schedule_merge_request_cleanup_refs”, “cronjob:schedule_migrate_external_diffs”, “cronjob:ssh_keys_expired_notification”, “cronjob:ssh_keys_expiring_soon_notification”, “cronjob:stuck_ci_jobs”, “cronjob:stuck_export_jobs”, “cronjob:stuck_merge_jobs”, “cronjob:trending_projects”, “cronjob:update_container_registry_info”, “cronjob:user_status_cleanup_batch”, “cronjob:users_create_statistics”, “cronjob:users_deactivate_dormant_users”, “cronjob:x509_issuer_crl_check”, “default”, “delete_diff_files”, “delete_merged_branches”, “delete_stored_files”, “delete_user”, “dependency_proxy:purge_dependency_proxy_cache”, “dependency_proxy_blob:dependency_proxy_cleanup_blob”, “dependency_proxy_manifest:dependency_proxy_cleanup_manifest”, “deployment:deployments_archive_in_project”, “deployment:deployments_drop_older_deployments”, “deployment:deployments_hooks”, “deployment:deployments_link_merge_request”, “deployment:deployments_update_environment”, “design_management_copy_design_collection”, “design_management_new_version”, “destroy_pages_deployments”, “detect_repository_languages”, “disallow_two_factor_for_group”, “disallow_two_factor_for_subgroups”, “email_receiver”, “emails_on_push”, “environments_auto_stop”, “environments_canary_ingress_update”, “error_tracking_issue_link”, “experiments_record_conversion_event”, “expire_build_instance_artifacts”, “export_csv”, “external_service_reactive_caching”, “file_hook”, “flush_counter_increments”, “gcp_cluster:cluster_configure_istio”, “gcp_cluster:cluster_install_app”, “gcp_cluster:cluster_patch_app”, “gcp_cluster:cluster_provision”, “gcp_cluster:cluster_update_app”, “gcp_cluster:cluster_upgrade_app”, “gcp_cluster:cluster_wait_for_app_installation”, “gcp_cluster:cluster_wait_for_app_update”, “gcp_cluster:cluster_wait_for_ingress_ip_address”, “gcp_cluster:clusters_applications_activate_service”, “gcp_cluster:clusters_applications_deactivate_service”, “gcp_cluster:clusters_applications_uninstall”, “gcp_cluster:clusters_applications_wait_for_uninstall_app”, “gcp_cluster:clusters_cleanup_project_namespace”, “gcp_cluster:clusters_cleanup_service_account”, “gcp_cluster:wait_for_cluster_creation”, “github_import_advance_stage”, “github_importer:github_import_import_diff_note”, “github_importer:github_import_import_issue”, “github_importer:github_import_import_lfs_object”, “github_importer:github_import_import_note”, “github_importer:github_import_import_pull_request”, “github_importer:github_import_import_pull_request_merged_by”, “github_importer:github_import_import_pull_request_review”, “github_importer:github_import_refresh_import_jid”, “github_importer:github_import_stage_finish_import”, “github_importer:github_import_stage_import_base_data”, “github_importer:github_import_stage_import_issues_and_diff_notes”, “github_importer:github_import_stage_import_lfs_objects”, “github_importer:github_import_stage_import_notes”, “github_importer:github_import_stage_import_pull_requests”, “github_importer:github_import_stage_import_pull_requests_merged_by”, “github_importer:github_import_stage_import_pull_requests_reviews”, “github_importer:github_import_stage_import_repository”, “gitlab_performance_bar_stats”, “gitlab_shell”, “group_destroy”, “group_export”, “group_import”, “hashed_storage:hashed_storage_migrator”, “hashed_storage:hashed_storage_project_migrate”, “hashed_storage:hashed_storage_project_rollback”, “hashed_storage:hashed_storage_rollbacker”, “import_issues_csv”, “incident_management:incident_management_add_severity_system_note”, “incident_management:incident_management_pager_duty_process_incident”, “incident_management:incident_management_process_alert_worker_v2”, “integrations_create_external_cross_reference”, “invalid_gpg_signature_update”, “irker”, “issuable_export_csv”, “issuable_label_links_destroy”, “issuables_clear_groups_issue_counter”, “issue_placement”, “issue_rebalancing”, “issues_placement”, “issues_rebalancing”, “jira_connect:jira_connect_forward_event”, “jira_connect:jira_connect_retry_request”, “jira_connect:jira_connect_sync_branch”, “jira_connect:jira_connect_sync_builds”, “jira_connect:jira_connect_sync_deployments”, “jira_connect:jira_connect_sync_feature_flags”, “jira_connect:jira_connect_sync_merge_request”, “jira_connect:jira_connect_sync_project”, “jira_importer:jira_import_advance_stage”, “jira_importer:jira_import_import_issue”, “jira_importer:jira_import_stage_finish_import”, “jira_importer:jira_import_stage_import_attachments”, “jira_importer:jira_import_stage_import_issues”, “jira_importer:jira_import_stage_import_labels”, “jira_importer:jira_import_stage_import_notes”, “jira_importer:jira_import_stage_start_import”, “mail_scheduler:mail_scheduler_issue_due”, “mail_scheduler:mail_scheduler_notification_service”, “mailers”, “merge”, “merge_request_cleanup_refs”, “merge_request_mergeability_check”, “merge_requests_delete_source_branch”, “merge_requests_handle_assignees_change”, “merge_requests_resolve_todos”, “metrics_dashboard_prune_old_annotations”, “metrics_dashboard_sync_dashboards”, “migrate_external_diffs”, “namespaceless_project_destroy”, “namespaces_invite_team_email”, “namespaces_onboarding_issue_created”, “namespaces_onboarding_pipeline_created”, “namespaces_onboarding_progress”, “namespaces_onboarding_user_added”, “namespaces_process_sync_events”, “new_issue”, “new_merge_request”, “new_note”, “object_pool:object_pool_create”, “object_pool:object_pool_destroy”, “object_pool:object_pool_join”, “object_pool:object_pool_schedule_join”, “object_storage:object_storage_background_move”, “object_storage:object_storage_migrate_uploads”, “package_repositories:packages_debian_generate_distribution”, “package_repositories:packages_debian_process_changes”, “package_repositories:packages_go_sync_packages”, “package_repositories:packages_helm_extraction”, “package_repositories:packages_maven_metadata_sync”, “package_repositories:packages_nuget_extraction”, “package_repositories:packages_rubygems_extraction”, “packages_composer_cache_update”, “pages”, “pages_domain_ssl_renewal”, “pages_domain_verification”, “pages_transfer”, “pages_update_configuration”, “phabricator_import_import_tasks”, “pipeline_background:archive_trace”, “pipeline_background:ci_archive_trace”, “pipeline_background:ci_build_trace_chunk_flush”, “pipeline_background:ci_daily_build_group_report_results”, “pipeline_background:ci_pending_builds_update_group”, “pipeline_background:ci_pending_builds_update_project”, “pipeline_background:ci_pipeline_artifacts_coverage_report”, “pipeline_background:ci_pipeline_artifacts_create_quality_report”, “pipeline_background:ci_pipeline_success_unlock_artifacts”, “pipeline_background:ci_ref_delete_unlock_artifacts”, “pipeline_background:ci_test_failure_history”, “pipeline_cache:expire_job_cache”, “pipeline_cache:expire_pipeline_cache”, “pipeline_creation:ci_external_pull_requests_create_pipeline”, “pipeline_creation:create_pipeline”, “pipeline_creation:merge_requests_create_pipeline”, “pipeline_creation:run_pipeline_schedule”, “pipeline_default:ci_create_cross_project_pipeline”, “pipeline_default:ci_create_downstream_pipeline”, “pipeline_default:ci_drop_pipeline”, “pipeline_default:ci_merge_requests_add_todo_when_build_fails”, “pipeline_default:ci_pipeline_bridge_status”, “pipeline_default:ci_retry_pipeline”, “pipeline_default:pipeline_metrics”, “pipeline_default:pipeline_notification”, “pipeline_hooks:build_hooks”, “pipeline_hooks:pipeline_hooks”, “pipeline_processing:build_finished”, “pipeline_processing:build_queue”, “pipeline_processing:build_success”, “pipeline_processing:ci_build_finished”, “pipeline_processing:ci_build_prepare”, “pipeline_processing:ci_build_schedule”, “pipeline_processing:ci_initial_pipeline_process”, “pipeline_processing:ci_resource_groups_assign_resource_from_resource_group”, “pipeline_processing:pipeline_process”, “pipeline_processing:stage_update”, “pipeline_processing:update_head_pipeline_for_merge_request”, “post_receive”, “process_commit”, “project_cache”, “project_daily_statistics”, “project_destroy”, “project_export”, “project_service”, “projects_git_garbage_collect”, “projects_post_creation”, “projects_process_sync_events”, “projects_schedule_bulk_repository_shard_moves”, “projects_update_repository_storage”, “prometheus_create_default_alerts”, “propagate_integration”, “propagate_integration_group”, “propagate_integration_inherit”, “propagate_integration_inherit_descendant”, “propagate_integration_project”, “reactive_caching”, “rebase”, “releases_create_evidence”, “remote_mirror_notification”, “repository_check:repository_check_batch”, “repository_check:repository_check_clear”, “repository_check:repository_check_single_repository”, “repository_cleanup”, “repository_fork”, “repository_import”, “repository_remove_remote”, “repository_update_remote_mirror”, “self_monitoring_project_create”, “self_monitoring_project_delete”, “service_desk_email_receiver”, “snippets_schedule_bulk_repository_shard_moves”, “snippets_update_repository_storage”, “system_hook_push”, “tasks_to_be_done_create”, “todos_destroyer:todos_destroyer_confidential_issue”, “todos_destroyer:todos_destroyer_destroyed_designs”, “todos_destroyer:todos_destroyer_destroyed_issuable”, “todos_destroyer:todos_destroyer_entity_leave”, “todos_destroyer:todos_destroyer_group_private”, “todos_destroyer:todos_destroyer_private_features”, “todos_destroyer:todos_destroyer_project_private”, “unassign_issuables:members_destroyer_unassign_issuables”, “update_external_pull_requests”, “update_highest_role”, “update_merge_requests”, “update_namespace_statistics:namespaces_root_statistics”, “update_namespace_statistics:namespaces_schedule_aggregation”, “update_project_statistics”, “upload_checksum”, “web_hook”, “web_hooks_destroy”, “web_hooks_log_execution”, “wikis_git_garbage_collect”, “x509_certificate_revoke”]
{“severity”:“INFO”,“time”:“2022-01-15T11:38:07.897Z”,“message”:“Booting Sidekiq 6.3.1 with redis options {:instrumentation_class=\u003eGitlab::Instrumentation::Redis::Queues, :path=\u003e"/var/opt/gitlab/redis/redis.socket", :namespace=\u003e"resque:gitlab"}”}
{“severity”:“INFO”,“time”:“2022-01-15T11:38:08.376Z”,“message”:“GitLab reliable fetch activated!”}
{“severity”:“INFO”,“time”:“2022-01-15T11:38:08.501Z”,“message”:“Cron Jobs - add job with name: stuck_ci_jobs_worker”}
{“severity”:“INFO”,“time”:“2022-01-15T11:38:08.502Z”,“message”:“Cron Jobs - add job with name: pipeline_schedule_worker”}
{“severity”:“INFO”,“time”:“2022-01-15T11:38:08.503Z”,“message”:“Cron Jobs - add job with name: expire_build_artifacts_worker”}
{“severity”:“INFO”,“time”:“2022-01-15T11:38:08.504Z”,“message”:“Cron Jobs - add job with name: ci_pipelines_expire_artifacts_worker”}
{“severity”:“INFO”,“time”:“2022-01-15T11:38:08.506Z”,“message”:“Cron Jobs - add job with name: ci_schedule_delete_objects_worker”}
{“severity”:“INFO”,“time”:“2022-01-15T11:38:08.507Z”,“message”:“Cron Jobs - add job with name: environments_auto_stop_cron_worker”}
{“severity”:“INFO”,“time”:“2022-01-15T11:38:08.508Z”,“message”:“Cron Jobs - add job with name: environments_auto_delete_cron_worker”}
{“severity”:“INFO”,“time”:“2022-01-15T11:38:08.509Z”,“message”:“Cron Jobs - add job with name: repository_check_worker”}
{“severity”:“INFO”,“time”:“2022-01-15T11:38:08.510Z”,“message”:“Cron Jobs - add job with name: admin_email_worker”}
{“severity”:“INFO”,“time”:“2022-01-15T11:38:08.511Z”,“message”:“Cron Jobs - add job with name: personal_access_tokens_expiring_worker”}
{“severity”:“INFO”,“time”:“2022-01-15T11:38:08.512Z”,“message”:“Cron Jobs - add job with name: personal_access_tokens_expired_notification_worker”}
{“severity”:“INFO”,“time”:“2022-01-15T11:38:08.513Z”,“message”:“Cron Jobs - add job with name: repository_archive_cache_worker”}
{“severity”:“INFO”,“time”:“2022-01-15T11:38:08.514Z”,“message”:“Cron Jobs - add job with name: import_export_project_cleanup_worker”}
{“severity”:“INFO”,“time”:“2022-01-15T11:38:08.515Z”,“message”:“Cron Jobs - add job with name: ci_archive_traces_cron_worker”}
{“severity”:“INFO”,“time”:“2022-01-15T11:38:08.516Z”,“message”:“Cron Jobs - add job with name: requests_profiles_worker”}
{“severity”:“INFO”,“time”:“2022-01-15T11:38:08.517Z”,“message”:“Cron Jobs - add job with name: remove_expired_members_worker”}
{“severity”:“INFO”,“time”:“2022-01-15T11:38:08.518Z”,“message”:“Cron Jobs - add job with name: remove_expired_group_links_worker”}
{“severity”:“INFO”,“time”:“2022-01-15T11:38:08.519Z”,“message”:“Cron Jobs - add job with name: remove_unaccepted_member_invites_worker”}
{“severity”:“INFO”,“time”:“2022-01-15T11:38:08.520Z”,“message”:“Cron Jobs - add job with name: prune_old_events_worker”}
{“severity”:“INFO”,“time”:“2022-01-15T11:38:08.521Z”,“message”:“Cron Jobs - add job with name: trending_projects_worker”}
{“severity”:“INFO”,“time”:“2022-01-15T11:38:08.522Z”,“message”:“Cron Jobs - add job with name: remove_unreferenced_lfs_objects_worker”}
{“severity”:“INFO”,“time”:“2022-01-15T11:38:08.523Z”,“message”:“Cron Jobs - add job with name: import_stuck_project_import_jobs”}
{“severity”:“INFO”,“time”:“2022-01-15T11:38:08.524Z”,“message”:“Cron Jobs - add job with name: jira_import_stuck_jira_import_jobs”}
{“severity”:“INFO”,“time”:“2022-01-15T11:38:08.524Z”,“message”:“Cron Jobs - add job with name: stuck_export_jobs_worker”}
{“severity”:“INFO”,“time”:“2022-01-15T11:38:08.525Z”,“message”:“Cron Jobs - add job with name: gitlab_service_ping_worker”}
{“severity”:“INFO”,“time”:“2022-01-15T11:38:08.526Z”,“message”:“Cron Jobs - add job with name: stuck_merge_jobs_worker”}
{“severity”:“INFO”,“time”:“2022-01-15T11:38:08.527Z”,“message”:“Cron Jobs - add job with name: pages_domain_verification_cron_worker”}
{“severity”:“INFO”,“time”:“2022-01-15T11:38:08.528Z”,“message”:“Cron Jobs - add job with name: pages_domain_removal_cron_worker”}
{“severity”:“INFO”,“time”:“2022-01-15T11:38:08.529Z”,“message”:“Cron Jobs - add job with name: pages_domain_ssl_renewal_cron_worker”}
{“severity”:“INFO”,“time”:“2022-01-15T11:38:08.530Z”,“message”:“Cron Jobs - add job with name: issue_due_scheduler_worker”}
{“severity”:“INFO”,“time”:“2022-01-15T11:38:08.531Z”,“message”:“Cron Jobs - add job with name: metrics_dashboard_schedule_annotations_prune_worker”}
{“severity”:“INFO”,“time”:“2022-01-15T11:38:08.532Z”,“message”:“Cron Jobs - add job with name: schedule_migrate_external_diffs_worker”}
{“severity”:“INFO”,“time”:“2022-01-15T11:38:08.532Z”,“message”:“Cron Jobs - add job with name: namespaces_prune_aggregation_schedules_worker”}
{“severity”:“INFO”,“time”:“2022-01-15T11:38:08.533Z”,“message”:“Cron Jobs - add job with name: container_expiration_policy_worker”}
{“severity”:“INFO”,“time”:“2022-01-15T11:38:08.534Z”,“message”:“Cron Jobs - add job with name: image_ttl_group_policy_worker”}
{“severity”:“INFO”,“time”:“2022-01-15T11:38:08.535Z”,“message”:“Cron Jobs - add job with name: cleanup_dependency_proxy_worker”}
{“severity”:“INFO”,“time”:“2022-01-15T11:38:08.536Z”,“message”:“Cron Jobs - add job with name: x509_issuer_crl_check_worker”}
{“severity”:“INFO”,“time”:“2022-01-15T11:38:08.537Z”,“message”:“Cron Jobs - add job with name: users_create_statistics_worker”}
{“severity”:“INFO”,“time”:“2022-01-15T11:38:08.538Z”,“message”:“Cron Jobs - add job with name: authorized_project_update_periodic_recalculate_worker”}
{“severity”:“INFO”,“time”:“2022-01-15T11:38:08.539Z”,“message”:“Cron Jobs - add job with name: update_container_registry_info_worker”}
{“severity”:“INFO”,“time”:“2022-01-15T11:38:08.540Z”,“message”:“Cron Jobs - add job with name: postgres_dynamic_partitions_manager”}
{“severity”:“INFO”,“time”:“2022-01-15T11:38:08.541Z”,“message”:“Cron Jobs - add job with name: postgres_dynamic_partitions_dropper”}
{“severity”:“INFO”,“time”:“2022-01-15T11:38:08.542Z”,“message”:“Cron Jobs - add job with name: ci_platform_metrics_update_cron_worker”}
{“severity”:“INFO”,“time”:“2022-01-15T11:38:08.543Z”,“message”:“Cron Jobs - add job with name: analytics_usage_trends_count_job_trigger_worker”}
{“severity”:“INFO”,“time”:“2022-01-15T11:38:08.544Z”,“message”:“Cron Jobs - add job with name: member_invitation_reminder_emails_worker”}
{“severity”:“INFO”,“time”:“2022-01-15T11:38:08.544Z”,“message”:“Cron Jobs - add job with name: schedule_merge_request_cleanup_refs_worker”}
{“severity”:“INFO”,“time”:“2022-01-15T11:38:08.545Z”,“message”:“Cron Jobs - add job with name: manage_evidence_worker”}
{“severity”:“INFO”,“time”:“2022-01-15T11:38:08.546Z”,“message”:“Cron Jobs - add job with name: user_status_cleanup_batch_worker”}
{“severity”:“INFO”,“time”:“2022-01-15T11:38:08.547Z”,“message”:“Cron Jobs - add job with name: ssh_keys_expired_notification_worker”}
{“severity”:“INFO”,“time”:“2022-01-15T11:38:08.548Z”,“message”:“Cron Jobs - add job with name: namespaces_in_product_marketing_emails_worker”}
{“severity”:“INFO”,“time”:“2022-01-15T11:38:08.549Z”,“message”:“Cron Jobs - add job with name: ssh_keys_expiring_soon_notification_worker”}
{“severity”:“INFO”,“time”:“2022-01-15T11:38:08.550Z”,“message”:“Cron Jobs - add job with name: users_deactivate_dormant_users_worker”}
{“severity”:“INFO”,“time”:“2022-01-15T11:38:08.551Z”,“message”:“Cron Jobs - add job with name: ci_delete_unit_tests_worker”}
{“severity”:“INFO”,“time”:“2022-01-15T11:38:08.552Z”,“message”:“Cron Jobs - add job with name: batched_background_migrations_worker”}
{“severity”:“INFO”,“time”:“2022-01-15T11:38:08.553Z”,“message”:“Cron Jobs - add job with name: issues_reschedule_stuck_issue_rebalances”}
{“severity”:“INFO”,“time”:“2022-01-15T11:38:08.553Z”,“message”:“Cron Jobs - add job with name: clusters_integrations_check_prometheus_health_worker”}
{“severity”:“WARN”,“time”:“2022-01-15T11:38:09.313Z”,“message”:“Creating scope :without_statuses. Overwriting existing method CommitStatus.without_statuses.”}
{“severity”:“WARN”,“time”:“2022-01-15T11:38:09.334Z”,“message”:“Creating scope :without_statuses. Overwriting existing method Ci::Pipeline.without_statuses.”}
{“severity”:“WARN”,“time”:“2022-01-15T11:38:09.414Z”,“message”:“Creating scope :in_pipelines. Overwriting existing method Ci::Build.in_pipelines.”}
{“severity”:“WARN”,“time”:“2022-01-15T11:38:09.582Z”,“message”:“Creating scope :of_projects. Overwriting existing method MergeRequest.of_projects.”}
{“severity”:“WARN”,“time”:“2022-01-15T11:38:09.583Z”,“message”:“Creating scope :join_project. Overwriting existing method MergeRequest.join_project.”}
{“severity”:“WARN”,“time”:“2022-01-15T11:38:09.583Z”,“message”:“Creating scope :references_project. Overwriting existing method MergeRequest.references_project.”}
{“severity”:“WARN”,“time”:“2022-01-15T11:38:09.595Z”,“message”:“Creating scope :system. Overwriting existing method Note.system.”}
{“severity”:“WARN”,“time”:“2022-01-15T11:38:09.868Z”,“message”:“Creating scope :order_created_desc. Overwriting existing method Packages::Package.order_created_desc.”}
{“severity”:“WARN”,“time”:“2022-01-15T11:38:09.868Z”,“message”:“Creating scope :order_name_desc. Overwriting existing method Packages::Package.order_name_desc.”}
{“severity”:“WARN”,“time”:“2022-01-15T11:38:10.662Z”,“message”:“Creating scope :open. Overwriting existing method AlertManagement::Alert.open.”}
{“severity”:“WARN”,“time”:“2022-01-15T11:38:10.878Z”,“message”:“Creating scope :order_name_asc. Overwriting existing method Label.order_name_asc.”}
{“severity”:“WARN”,“time”:“2022-01-15T11:38:10.878Z”,“message”:“Creating scope :order_name_desc. Overwriting existing method Label.order_name_desc.”}
{“severity”:“WARN”,“time”:“2022-01-15T11:38:11.976Z”,“message”:“Creating scope :open. Overwriting existing method ExternalPullRequest.open.”}
{“severity”:“WARN”,“time”:“2022-01-15T11:38:12.059Z”,“message”:“Creating scope :system. Overwriting existing method PrometheusMetric.system.”}
{“severity”:“WARN”,“time”:“2022-01-15T11:38:12.121Z”,“message”:“Creating scope :order_name_asc. Overwriting existing method Member.order_name_asc.”}
{“severity”:“WARN”,“time”:“2022-01-15T11:38:12.121Z”,“message”:“Creating scope :order_name_desc. Overwriting existing method Member.order_name_desc.”}
{“severity”:“WARN”,“time”:“2022-01-15T11:38:12.170Z”,“message”:“Creating scope :with_user. Overwriting existing method GroupMember.with_user.”}
{“severity”:“WARN”,“time”:“2022-01-15T11:38:12.467Z”,“message”:“Creating scope :database. Overwriting existing method Ci::BuildTraceChunk.database.”}
{“severity”:“WARN”,“time”:“2022-01-15T11:38:12.474Z”,“message”:“Creating scope :without_statuses. Overwriting existing method Ci::Stage.without_statuses.”}
{“severity”:“WARN”,“time”:“2022-01-15T11:38:12.474Z”,“message”:“Creating scope :created. Overwriting existing method Ci::Stage.created.”}
{“severity”:“WARN”,“time”:“2022-01-15T11:38:12.474Z”,“message”:“Creating scope :pending. Overwriting existing method Ci::Stage.pending.”}
{“severity”:“WARN”,“time”:“2022-01-15T11:38:12.475Z”,“message”:“Creating scope :running. Overwriting existing method Ci::Stage.running.”}
{“severity”:“WARN”,“time”:“2022-01-15T11:38:12.475Z”,“message”:“Creating scope :success. Overwriting existing method Ci::Stage.success.”}
{“severity”:“WARN”,“time”:“2022-01-15T11:38:12.475Z”,“message”:“Creating scope :failed. Overwriting existing method Ci::Stage.failed.”}
{“severity”:“WARN”,“time”:“2022-01-15T11:38:12.475Z”,“message”:“Creating scope :canceled. Overwriting existing method Ci::Stage.canceled.”}
{“severity”:“WARN”,“time”:“2022-01-15T11:38:12.475Z”,“message”:“Creating scope :skipped. Overwriting existing method Ci::Stage.skipped.”}
{“severity”:“WARN”,“time”:“2022-01-15T11:38:12.475Z”,“message”:“Creating scope :manual. Overwriting existing method Ci::Stage.manual.”}
{“severity”:“WARN”,“time”:“2022-01-15T11:38:12.475Z”,“message”:“Creating scope :scheduled. Overwriting existing method Ci::Stage.scheduled.”}
{“severity”:“WARN”,“time”:“2022-01-15T11:38:12.475Z”,“message”:“Creating scope :preparing. Overwriting existing method Ci::Stage.preparing.”}
{“severity”:“WARN”,“time”:“2022-01-15T11:38:12.476Z”,“message”:“Creating scope :waiting_for_resource. Overwriting existing method Ci::Stage.waiting_for_resource.”}
{“severity”:“WARN”,“time”:“2022-01-15T11:38:14.515Z”,“message”:“Creating scope :system. Overwriting existing method Gitlab::DatabaseImporters::CommonMetrics::PrometheusMetric.system.”}
{“severity”:“INFO”,“time”:“2022-01-15T11:38:18.877Z”,“message”:“Booted Rails 6.1.4.1 application in production environment”}
{“severity”:“INFO”,“time”:“2022-01-15T11:38:18.877Z”,“message”:“Running in ruby 2.7.5p203 (2021-11-24 revision f69aeb8314) [x86_64-linux]”}
{“severity”:“INFO”,“time”:“2022-01-15T11:38:18.877Z”,“message”:“See LICENSE and the LGPL-3.0 for licensing details.”}
{“severity”:“INFO”,“time”:“2022-01-15T11:38:18.877Z”,“message”:“Upgrade to Sidekiq Pro for more features and support: https://sidekiq.org”}
{“severity”:“INFO”,“time”:“2022-01-15T11:38:23.835Z”,“message”:“A worker terminated, shutting down the cluster”}
{“severity”:“INFO”,“time”:“2022-01-15T11:38:24.538Z”,“message”:“Starting cluster with 1 processes”}
I would try running:
gitlab-ctl reconfigure
and see what happens after this. Normally in the information after an upgrade it says if Gitlab isn’t behaving properly to run reconfigure and restart Gitlab to see if that helps. Have you tried that already or not yet?
I have already reconfigured and restarted GitLab. Unfortunately it doesn’t help. Thanks for the suggestion though.
It seems that somebody has already encountered a similar issue: Sidekiq Cluster keeps terminating - #3 by Yannis
Changing the port of puma on gitlab.rb fixed this issue.