Weird issues with build pipeline after upgrading to 14.6.0-ee

Hi everyone,

I am wondering if anyone else experienced weird issues after upgrading to 14.6.0. Since then our build pipeline has all sorts of strange issues which I will try to describe in this post.

When we create a new MR based on a branch that was fine before this happens:

As you can see “Checking pipeline status” spins there forever (I can confirm there is nothing wrong with gitlab-ci.yml whatsoever). This happens with any MR on any branch in any of our projects now.

Second issue is that pending pipelines are stuck in a weird state like this:

The pending (orange) state should actually say “failed”. Instead it keeps being like this forever. You cannot restart the pipeline, but you can restart a single job (which does not change anything there).

Another problem I see is when I try to create a new project from a template (no matter which one). It keeps doing this forever and the bundle process goes full power until I delete the project.

Screenshot 2021-12-23 at 13.37.46

I am running GitLab on a self-hosted Linux machine and would describe myself as an experienced DevOps which is why I would eliminate typical issues like “low disk space”. Unfortunately I have not been able to find anything useful in the logs.

I even tried a full re-install with omnibus and restoring one of our backups - problems persist. Is there something else I could try? I am very confused by the issues I see because they just don’t make sense.

Thanks :slight_smile:
Amadeus

I digged deeper into the logs and it seems sidekiq has some problems (see log attached).

What could be the reason for this / how can I debug this further?

{"severity":"INFO","time":"2021-12-24T09:27:19.092Z","message":"A worker terminated, shutting down the cluster"}
{"severity":"INFO","time":"2021-12-24T09:27:20.165Z","message":"Starting cluster with 1 processes"}
2021-12-24T09:27:35.221Z pid=16459 tid=5mj INFO: Listening on queues ["adjourned_project_deletion", "admin_emails", "analytics_code_review_metrics", "analytics_cycle_analytics_group_data_loader", "analytics_devops_adoption_create_snapshot", "analytics_usage_trends_counter_job", "app_sec_dast_scans_consistency", "approval_rules_external_approval_rule_payload", "approve_blocked_pending_approval_users", "audit_events_audit_event_streaming", "authorized_keys", "authorized_project_update:authorized_project_update_project_create", "authorized_project_update:authorized_project_update_project_group_link_create", "authorized_project_update:authorized_project_update_project_recalculate", "authorized_project_update:authorized_project_update_project_recalculate_per_user", "authorized_project_update:authorized_project_update_user_refresh_from_replica", "authorized_project_update:authorized_project_update_user_refresh_over_user_range", "authorized_project_update:authorized_project_update_user_refresh_with_low_urgency", "authorized_projects", "auto_devops:auto_devops_disable", "auto_merge:auto_merge_process", "auto_merge:merge_trains_refresh", "background_migration", "bulk_import", "bulk_imports_entity", "bulk_imports_export_request", "bulk_imports_pipeline", "bulk_imports_relation_export", "chaos:chaos_cpu_spin", "chaos:chaos_db_spin", "chaos:chaos_kill", "chaos:chaos_leak_mem", "chaos:chaos_sleep", "chat_notification", "ci_batch_reset_minutes", "ci_delete_objects", "ci_upstream_projects_subscriptions_cleanup", "container_repository:cleanup_container_repository", "container_repository:container_expiration_policies_cleanup_container_repository", "container_repository:delete_container_repository", "create_commit_signature", "create_github_webhook", "create_note_diff_file", "cronjob:active_user_count_threshold", "cronjob:adjourned_group_deletion", "cronjob:adjourned_projects_deletion_cron", "cronjob:admin_email", "cronjob:analytics_devops_adoption_create_all_snapshots", "cronjob:analytics_usage_trends_count_job_trigger", "cronjob:app_sec_dast_profile_schedule", "cronjob:authorized_project_update_periodic_recalculate", "cronjob:ci_archive_traces_cron", "cronjob:ci_delete_unit_tests", "cronjob:ci_pipeline_artifacts_expire_artifacts", "cronjob:ci_platform_metrics_update_cron", "cronjob:ci_schedule_delete_objects_cron", "cronjob:ci_stuck_builds_drop_running", "cronjob:ci_stuck_builds_drop_scheduled", "cronjob:clear_shared_runners_minutes", "cronjob:clusters_integrations_check_prometheus_health", "cronjob:container_expiration_policy", "cronjob:database_batched_background_migration", "cronjob:database_drop_detached_partitions", "cronjob:database_partition_management", "cronjob:dependency_proxy_image_ttl_group_policy", "cronjob:elastic_cluster_reindexing_cron", "cronjob:elastic_index_bulk_cron", "cronjob:elastic_index_initial_bulk_cron", "cronjob:elastic_migration", "cronjob:elastic_remove_expired_namespace_subscriptions_from_index_cron", "cronjob:environments_auto_delete_cron", "cronjob:environments_auto_stop_cron", "cronjob:expire_build_artifacts", "cronjob:geo_container_repository_sync_dispatch", "cronjob:geo_file_download_dispatch", "cronjob:geo_metrics_update", "cronjob:geo_prune_event_log", "cronjob:geo_registry_sync", "cronjob:geo_repository_sync", "cronjob:geo_repository_verification_primary_batch", "cronjob:geo_repository_verification_secondary_scheduler", "cronjob:geo_repository_verification_secondary_shard", "cronjob:geo_scheduler_per_shard_scheduler", "cronjob:geo_scheduler_primary_per_shard_scheduler", "cronjob:geo_scheduler_secondary_per_shard_scheduler", "cronjob:geo_secondary_registry_consistency", "cronjob:geo_secondary_usage_data_cron", "cronjob:geo_sidekiq_cron_config", "cronjob:geo_sync_timeout_cron", "cronjob:geo_verification_cron", "cronjob:gitlab_service_ping", "cronjob:historical_data", "cronjob:import_export_project_cleanup", "cronjob:import_software_licenses", "cronjob:import_stuck_project_import_jobs", "cronjob:incident_management_incident_sla_exceeded_check", "cronjob:incident_management_oncall_rotations_persist_all_rotations_shifts_job", "cronjob:incident_management_pending_escalations_schedule_check_cron", "cronjob:issue_due_scheduler", "cronjob:issues_reschedule_stuck_issue_rebalances", "cronjob:iterations_cadences_create_iterations", "cronjob:iterations_cadences_schedule_create_iterations", "cronjob:iterations_update_status", "cronjob:jira_import_stuck_jira_import_jobs", "cronjob:ldap_all_groups_sync", "cronjob:ldap_sync", "cronjob:loose_foreign_keys_cleanup", "cronjob:member_invitation_reminder_emails", "cronjob:metrics_dashboard_schedule_annotations_prune", "cronjob:namespaces_in_product_marketing_emails", "cronjob:namespaces_prune_aggregation_schedules", "cronjob:network_policy_metrics", "cronjob:packages_composer_cache_cleanup", "cronjob:pages_domain_removal_cron", "cronjob:pages_domain_ssl_renewal_cron", "cronjob:pages_domain_verification_cron", "cronjob:partition_creation", "cronjob:personal_access_tokens_expired_notification", "cronjob:personal_access_tokens_expiring", "cronjob:pipeline_schedule", "cronjob:prune_old_events", "cronjob:pseudonymizer", "cronjob:releases_manage_evidence", "cronjob:remove_expired_group_links", "cronjob:remove_expired_members", "cronjob:remove_unaccepted_member_invites", "cronjob:remove_unreferenced_lfs_objects", "cronjob:repository_archive_cache", "cronjob:repository_check_dispatch", "cronjob:requests_profiles", "cronjob:schedule_merge_request_cleanup_refs", "cronjob:schedule_migrate_external_diffs", "cronjob:security_create_orchestration_policy", "cronjob:security_orchestration_policy_rule_schedule", "cronjob:ssh_keys_expired_notification", "cronjob:ssh_keys_expiring_soon_notification", "cronjob:stuck_ci_jobs", "cronjob:stuck_export_jobs", "cronjob:stuck_merge_jobs", "cronjob:sync_seat_link", "cronjob:trending_projects", "cronjob:update_all_mirrors", "cronjob:update_container_registry_info", "cronjob:update_max_seats_used_for_gitlab_com_subscriptions", "cronjob:user_status_cleanup_batch", "cronjob:users_create_statistics", "cronjob:users_deactivate_dormant_users", "cronjob:vulnerabilities_historical_statistics_deletion", "cronjob:vulnerabilities_statistics_schedule", "cronjob:x509_issuer_crl_check", "dast_site_validation", "default", "delete_diff_files", "delete_merged_branches", "delete_stored_files", "delete_user", "dependency_proxy:purge_dependency_proxy_cache", "dependency_proxy_blob:dependency_proxy_cleanup_blob", "dependency_proxy_manifest:dependency_proxy_cleanup_manifest", "deployment:deployments_archive_in_project", "deployment:deployments_auto_rollback", "deployment:deployments_drop_older_deployments", "deployment:deployments_hooks", "deployment:deployments_link_merge_request", "deployment:deployments_update_environment", "design_management_copy_design_collection", "design_management_new_version", "destroy_pages_deployments", "detect_repository_languages", "disallow_two_factor_for_group", "disallow_two_factor_for_subgroups", "dora_metrics:dora_daily_metrics_refresh", "elastic_association_indexer", "elastic_commit_indexer", "elastic_delete_project", "elastic_full_index", "elastic_indexing_control", "elastic_namespace_indexer", "elastic_namespace_rollout", "email_receiver", "emails_on_push", "environments_auto_stop", "environments_canary_ingress_update", "epics:epics_update_epics_dates", "epics_new_epic_issue", "error_tracking_issue_link", "experiments_record_conversion_event", "expire_build_instance_artifacts", "export_csv", "external_service_reactive_caching", "file_hook", "flush_counter_increments", "gcp_cluster:cluster_configure_istio", "gcp_cluster:cluster_install_app", "gcp_cluster:cluster_patch_app", "gcp_cluster:cluster_provision", "gcp_cluster:cluster_update_app", "gcp_cluster:cluster_upgrade_app", "gcp_cluster:cluster_wait_for_app_installation", "gcp_cluster:cluster_wait_for_app_update", "gcp_cluster:cluster_wait_for_ingress_ip_address", "gcp_cluster:clusters_applications_activate_service", "gcp_cluster:clusters_applications_deactivate_service", "gcp_cluster:clusters_applications_uninstall", "gcp_cluster:clusters_applications_wait_for_uninstall_app", "gcp_cluster:clusters_cleanup_project_namespace", "gcp_cluster:clusters_cleanup_service_account", "gcp_cluster:wait_for_cluster_creation", "geo:geo_batch_event_create", "geo:geo_batch_project_registry", "geo:geo_batch_project_registry_scheduler", "geo:geo_container_repository_sync", "geo:geo_design_repository_shard_sync", "geo:geo_design_repository_sync", "geo:geo_destroy", "geo:geo_event", "geo:geo_file_download", "geo:geo_file_registry_removal", "geo:geo_file_removal", "geo:geo_hashed_storage_attachments_migration", "geo:geo_hashed_storage_migration", "geo:geo_project_sync", "geo:geo_rename_repository", "geo:geo_repositories_clean_up", "geo:geo_repository_cleanup", "geo:geo_repository_destroy", "geo:geo_repository_shard_sync", "geo:geo_repository_verification_primary_shard", "geo:geo_repository_verification_primary_single", "geo:geo_repository_verification_secondary_single", "geo:geo_reverification_batch", "geo:geo_scheduler_primary_scheduler", "geo:geo_scheduler_scheduler", "geo:geo_scheduler_secondary_scheduler", "geo:geo_verification", "geo:geo_verification_batch", "geo:geo_verification_state_backfill", "geo:geo_verification_timeout", "github_import_advance_stage", "github_importer:github_import_import_diff_note", "github_importer:github_import_import_issue", "github_importer:github_import_import_lfs_object", "github_importer:github_import_import_note", "github_importer:github_import_import_pull_request", "github_importer:github_import_import_pull_request_merged_by", "github_importer:github_import_import_pull_request_review", "github_importer:github_import_refresh_import_jid", "github_importer:github_import_stage_finish_import", "github_importer:github_import_stage_import_base_data", "github_importer:github_import_stage_import_issues_and_diff_notes", "github_importer:github_import_stage_import_lfs_objects", "github_importer:github_import_stage_import_notes", "github_importer:github_import_stage_import_pull_requests", "github_importer:github_import_stage_import_pull_requests_merged_by", "github_importer:github_import_stage_import_pull_requests_reviews", "github_importer:github_import_stage_import_repository", "gitlab_performance_bar_stats", "gitlab_shell", "group_destroy", "group_export", "group_import", "group_saml_group_sync", "group_wikis_git_garbage_collect", "groups_create_event", "groups_export_memberships", "groups_schedule_bulk_repository_shard_moves", "groups_update_repository_storage", "groups_update_statistics", "hashed_storage:hashed_storage_migrator", "hashed_storage:hashed_storage_project_migrate", "hashed_storage:hashed_storage_project_rollback", "hashed_storage:hashed_storage_rollbacker", "import_issues_csv", "incident_management:incident_management_add_severity_system_note", "incident_management:incident_management_pager_duty_process_incident", "incident_management:incident_management_process_alert_worker_v2", "incident_management_apply_incident_sla_exceeded_label", "incident_management_oncall_rotations_persist_shifts_job", "incident_management_pending_escalations_alert_check", "incident_management_pending_escalations_alert_create", "incident_management_pending_escalations_issue_check", "integrations_create_external_cross_reference", "invalid_gpg_signature_update", "irker", "issuable_export_csv", "issuable_label_links_destroy", "issuables_clear_groups_issue_counter", "issue_placement", "issue_rebalancing", "issues_placement", "issues_rebalancing", "iterations:iterations_roll_over_issues", "jira_connect:jira_connect_forward_event", "jira_connect:jira_connect_retry_request", "jira_connect:jira_connect_sync_branch", "jira_connect:jira_connect_sync_builds", "jira_connect:jira_connect_sync_deployments", "jira_connect:jira_connect_sync_feature_flags", "jira_connect:jira_connect_sync_merge_request", "jira_connect:jira_connect_sync_project", "jira_importer:jira_import_advance_stage", "jira_importer:jira_import_import_issue", "jira_importer:jira_import_stage_finish_import", "jira_importer:jira_import_stage_import_attachments", "jira_importer:jira_import_stage_import_issues", "jira_importer:jira_import_stage_import_labels", "jira_importer:jira_import_stage_import_notes", "jira_importer:jira_import_stage_start_import", "ldap_group_sync", "mail_scheduler:mail_scheduler_issue_due", "mail_scheduler:mail_scheduler_notification_service", "mailers", "merge", "merge_request_cleanup_refs", "merge_request_mergeability_check", "merge_request_reset_approvals", "merge_requests_delete_source_branch", "merge_requests_handle_assignees_change", "merge_requests_resolve_todos", "merge_requests_sync_code_owner_approval_rules", "metrics_dashboard_prune_old_annotations", "metrics_dashboard_sync_dashboards", "migrate_external_diffs", "namespaceless_project_destroy", "namespaces_invite_team_email", "namespaces_onboarding_issue_created", "namespaces_onboarding_pipeline_created", "namespaces_onboarding_progress", "namespaces_onboarding_user_added", "namespaces_process_sync_events", "namespaces_sync_namespace_name", "new_epic", "new_issue", "new_merge_request", "new_note", "object_pool:object_pool_create", "object_pool:object_pool_destroy", "object_pool:object_pool_join", "object_pool:object_pool_schedule_join", "object_storage:object_storage_background_move", "object_storage:object_storage_migrate_uploads", "package_repositories:packages_debian_generate_distribution", "package_repositories:packages_debian_process_changes", "package_repositories:packages_go_sync_packages", "package_repositories:packages_helm_extraction", "package_repositories:packages_maven_metadata_sync", "package_repositories:packages_nuget_extraction", "package_repositories:packages_rubygems_extraction", "packages_composer_cache_update", "pages", "pages_domain_ssl_renewal", "pages_domain_verification", "pages_transfer", "pages_update_configuration", "personal_access_tokens:personal_access_tokens_groups_policy", "personal_access_tokens:personal_access_tokens_instance_policy", "phabricator_import_import_tasks", "pipeline_background:archive_trace", "pipeline_background:ci_archive_trace", "pipeline_background:ci_build_trace_chunk_flush", "pipeline_background:ci_daily_build_group_report_results", "pipeline_background:ci_minutes_update_project_and_namespace_usage", "pipeline_background:ci_pending_builds_update_group", "pipeline_background:ci_pending_builds_update_project", "pipeline_background:ci_pipeline_artifacts_coverage_report", "pipeline_background:ci_pipeline_artifacts_create_quality_report", "pipeline_background:ci_pipeline_success_unlock_artifacts", "pipeline_background:ci_ref_delete_unlock_artifacts", "pipeline_background:ci_sync_reports_to_report_approval_rules", "pipeline_background:ci_test_failure_history", "pipeline_cache:expire_job_cache", "pipeline_cache:expire_pipeline_cache", "pipeline_creation:ci_external_pull_requests_create_pipeline", "pipeline_creation:create_pipeline", "pipeline_creation:merge_requests_create_pipeline", "pipeline_creation:run_pipeline_schedule", "pipeline_default:ci_create_cross_project_pipeline", "pipeline_default:ci_create_downstream_pipeline", "pipeline_default:ci_drop_pipeline", "pipeline_default:ci_merge_requests_add_todo_when_build_fails", "pipeline_default:ci_pipeline_bridge_status", "pipeline_default:ci_retry_pipeline", "pipeline_default:ci_trigger_downstream_subscriptions", "pipeline_default:pipeline_metrics", "pipeline_default:pipeline_notification", "pipeline_hooks:build_hooks", "pipeline_hooks:pipeline_hooks", "pipeline_processing:build_finished", "pipeline_processing:build_queue", "pipeline_processing:build_success", "pipeline_processing:ci_build_finished", "pipeline_processing:ci_build_prepare", "pipeline_processing:ci_build_schedule", "pipeline_processing:ci_initial_pipeline_process", "pipeline_processing:ci_resource_groups_assign_resource_from_resource_group", "pipeline_processing:pipeline_process", "pipeline_processing:stage_update", "pipeline_processing:update_head_pipeline_for_merge_request", "post_receive", "process_commit", "project_cache", "project_daily_statistics", "project_destroy", "project_export", "project_import_schedule", "project_service", "project_template_export", "projects_git_garbage_collect", "projects_post_creation", "projects_process_sync_events", "projects_schedule_bulk_repository_shard_moves", "projects_update_repository_storage", "prometheus_create_default_alerts", "propagate_integration", "propagate_integration_group", "propagate_integration_inherit", "propagate_integration_inherit_descendant", "propagate_integration_project", "reactive_caching", "rebase", "refresh_license_compliance_checks", "releases_create_evidence", "remote_mirror_notification", "repository_check:repository_check_batch", "repository_check:repository_check_clear", "repository_check:repository_check_single_repository", "repository_cleanup", "repository_fork", "repository_import", "repository_push_audit_event", "repository_remove_remote", "repository_update_mirror", "repository_update_remote_mirror", "requirements_management_import_requirements_csv", "requirements_management_process_requirements_reports", "security_auto_fix", "security_scans:scan_security_report_secrets", "security_scans:security_store_scans", "security_scans:security_track_secure_scans", "security_scans:store_security_reports", "self_monitoring_project_create", "self_monitoring_project_delete", "service_desk_email_receiver", "set_user_status_based_on_user_cap_setting", "snippets_schedule_bulk_repository_shard_moves", "snippets_update_repository_storage", "status_page_publish", "sync_seat_link_request", "system_hook_push", "tasks_to_be_done_create", "todos_destroyer:todos_destroyer_confidential_epic", "todos_destroyer:todos_destroyer_confidential_issue", "todos_destroyer:todos_destroyer_destroyed_designs", "todos_destroyer:todos_destroyer_destroyed_issuable", "todos_destroyer:todos_destroyer_entity_leave", "todos_destroyer:todos_destroyer_group_private", "todos_destroyer:todos_destroyer_private_features", "todos_destroyer:todos_destroyer_project_private", "unassign_issuables:members_destroyer_unassign_issuables", "update_external_pull_requests", "update_highest_role", "update_merge_requests", "update_namespace_statistics:namespaces_root_statistics", "update_namespace_statistics:namespaces_schedule_aggregation", "update_project_statistics", "upload_checksum", "vulnerabilities_statistics_adjustment", "vulnerability_exports_export", "vulnerability_exports_export_deletion", "web_hook", "web_hooks_destroy", "web_hooks_log_execution", "wikis_git_garbage_collect", "x509_certificate_revoke"]
{"severity":"INFO","time":"2021-12-24T09:27:35.224Z","message":"Booting Sidekiq 6.3.1 with redis options {:instrumentation_class=\u003eGitlab::Instrumentation::Redis::Queues, :path=\u003e\"/var/opt/gitlab/redis/redis.socket\", :namespace=\u003e\"resque:gitlab\"}"}
{"severity":"INFO","time":"2021-12-24T09:27:38.096Z","message":"GitLab reliable fetch activated!"}
{"severity":"INFO","time":"2021-12-24T09:27:38.226Z","message":"Cron Jobs - deleted job with name: update_all_mirrors_worker"}
{"severity":"INFO","time":"2021-12-24T09:27:38.228Z","message":"Cron Jobs - deleted job with name: geo_sidekiq_cron_config_worker"}
{"severity":"INFO","time":"2021-12-24T09:27:38.230Z","message":"Cron Jobs - add job with name: stuck_ci_jobs_worker"}
{"severity":"INFO","time":"2021-12-24T09:27:38.232Z","message":"Cron Jobs - add job with name: pipeline_schedule_worker"}
[...]
{"severity":"WARN","time":"2021-12-24T09:27:40.052Z","message":"Creating scope :without_statuses. Overwriting existing method CommitStatus.without_statuses."}
{"severity":"WARN","time":"2021-12-24T09:27:40.102Z","message":"Creating scope :without_statuses. Overwriting existing method Ci::Pipeline.without_statuses."}
{"severity":"WARN","time":"2021-12-24T09:27:40.302Z","message":"Creating scope :in_pipelines. Overwriting existing method Ci::Build.in_pipelines."}
{"severity":"WARN","time":"2021-12-24T09:27:40.945Z","message":"Creating scope :of_projects. Overwriting existing method MergeRequest.of_projects."}
{"severity":"WARN","time":"2021-12-24T09:27:40.946Z","message":"Creating scope :join_project. Overwriting existing method MergeRequest.join_project."}
{"severity":"WARN","time":"2021-12-24T09:27:40.947Z","message":"Creating scope :references_project. Overwriting existing method MergeRequest.references_project."}
{"severity":"WARN","time":"2021-12-24T09:27:40.986Z","message":"Creating scope :system. Overwriting existing method Note.system."}
{"severity":"WARN","time":"2021-12-24T09:27:41.348Z","message":"Creating scope :closed. Overwriting existing method Iteration.closed."}
{"severity":"WARN","time":"2021-12-24T09:27:41.348Z","message":"Creating scope :within_timeframe. Overwriting existing method Iteration.within_timeframe."}
{"severity":"WARN","time":"2021-12-24T09:27:41.720Z","message":"Creating scope :opened. Overwriting existing method Epic.opened."}
{"severity":"WARN","time":"2021-12-24T09:27:41.720Z","message":"Creating scope :closed. Overwriting existing method Epic.closed."}
{"severity":"WARN","time":"2021-12-24T09:27:41.922Z","message":"Creating scope :order_created_desc. Overwriting existing method Packages::Package.order_created_desc."}
{"severity":"WARN","time":"2021-12-24T09:27:41.922Z","message":"Creating scope :order_name_desc. Overwriting existing method Packages::Package.order_name_desc."}
{"severity":"WARN","time":"2021-12-24T09:27:43.781Z","message":"Creating scope :open. Overwriting existing method AlertManagement::Alert.open."}
{"severity":"WARN","time":"2021-12-24T09:27:43.857Z","message":"Creating scope :with_associations. Overwriting existing method Board.with_associations."}
{"severity":"WARN","time":"2021-12-24T09:27:43.929Z","message":"Creating scope :movable. Overwriting existing method Boards::EpicList.movable."}
{"severity":"WARN","time":"2021-12-24T09:27:44.292Z","message":"Creating scope :order_name_asc. Overwriting existing method Label.order_name_asc."}
{"severity":"WARN","time":"2021-12-24T09:27:44.293Z","message":"Creating scope :order_name_desc. Overwriting existing method Label.order_name_desc."}
{"severity":"WARN","time":"2021-12-24T09:27:47.207Z","message":"Creating scope :with_logging_info. Overwriting existing method PagesDomain.with_logging_info."}
{"severity":"WARN","time":"2021-12-24T09:27:47.236Z","message":"Creating scope :system. Overwriting existing method PrometheusMetric.system."}
{"severity":"WARN","time":"2021-12-24T09:27:47.432Z","message":"Creating scope :open. Overwriting existing method ExternalPullRequest.open."}
{"severity":"WARN","time":"2021-12-24T09:27:47.498Z","message":"Creating scope :order_name_asc. Overwriting existing method Member.order_name_asc."}
{"severity":"WARN","time":"2021-12-24T09:27:47.499Z","message":"Creating scope :order_name_desc. Overwriting existing method Member.order_name_desc."}
{"severity":"WARN","time":"2021-12-24T09:27:47.678Z","message":"Creating scope :with_user. Overwriting existing method GroupMember.with_user."}
{"severity":"WARN","time":"2021-12-24T09:27:47.680Z","message":"Creating scope :reporters. Overwriting existing method GroupMember.reporters."}
{"severity":"WARN","time":"2021-12-24T09:27:47.680Z","message":"Creating scope :guests. Overwriting existing method GroupMember.guests."}
{"severity":"WARN","time":"2021-12-24T09:27:47.943Z","message":"Creating scope :extended. Overwriting existing method GitlabSubscription.extended."}
{"severity":"WARN","time":"2021-12-24T09:27:48.370Z","message":"Creating scope :without_statuses. Overwriting existing method Ci::Stage.without_statuses."}
{"severity":"WARN","time":"2021-12-24T09:27:48.370Z","message":"Creating scope :created. Overwriting existing method Ci::Stage.created."}
{"severity":"WARN","time":"2021-12-24T09:27:48.371Z","message":"Creating scope :pending. Overwriting existing method Ci::Stage.pending."}
{"severity":"WARN","time":"2021-12-24T09:27:48.371Z","message":"Creating scope :running. Overwriting existing method Ci::Stage.running."}
{"severity":"WARN","time":"2021-12-24T09:27:48.371Z","message":"Creating scope :success. Overwriting existing method Ci::Stage.success."}
{"severity":"WARN","time":"2021-12-24T09:27:48.372Z","message":"Creating scope :failed. Overwriting existing method Ci::Stage.failed."}
{"severity":"WARN","time":"2021-12-24T09:27:48.372Z","message":"Creating scope :canceled. Overwriting existing method Ci::Stage.canceled."}
{"severity":"WARN","time":"2021-12-24T09:27:48.373Z","message":"Creating scope :skipped. Overwriting existing method Ci::Stage.skipped."}
{"severity":"WARN","time":"2021-12-24T09:27:48.373Z","message":"Creating scope :manual. Overwriting existing method Ci::Stage.manual."}
{"severity":"WARN","time":"2021-12-24T09:27:48.373Z","message":"Creating scope :scheduled. Overwriting existing method Ci::Stage.scheduled."}
{"severity":"WARN","time":"2021-12-24T09:27:48.374Z","message":"Creating scope :preparing. Overwriting existing method Ci::Stage.preparing."}
{"severity":"WARN","time":"2021-12-24T09:27:48.374Z","message":"Creating scope :waiting_for_resource. Overwriting existing method Ci::Stage.waiting_for_resource."}
{"severity":"WARN","time":"2021-12-24T09:27:48.442Z","message":"Creating scope :database. Overwriting existing method Ci::BuildTraceChunk.database."}
{"severity":"WARN","time":"2021-12-24T09:27:49.555Z","message":"Creating scope :approved_by_committer. Overwriting existing method MergeRequests::ComplianceViolation.approved_by_committer."}
{"severity":"WARN","time":"2021-12-24T09:27:49.559Z","message":"Creating scope :runnable_schedules. Overwriting existing method Security::OrchestrationPolicyRuleSchedule.runnable_schedules."}
{"severity":"WARN","time":"2021-12-24T09:27:50.676Z","message":"Creating scope :regular. Overwriting existing method EE::Gitlab::BackgroundMigration::MigrateApproverToApprovalRules::ApprovalMergeRequestRule.regular."}
{"severity":"WARN","time":"2021-12-24T09:27:50.677Z","message":"Creating scope :code_owner. Overwriting existing method EE::Gitlab::BackgroundMigration::MigrateApproverToApprovalRules::ApprovalMergeRequestRule.code_owner."}
{"severity":"WARN","time":"2021-12-24T09:27:53.223Z","message":"Creating scope :system. Overwriting existing method Gitlab::DatabaseImporters::CommonMetrics::PrometheusMetric.system."}
{"severity":"INFO","time":"2021-12-24T09:28:09.111Z","message":"Booted Rails 6.1.4.1 application in production environment"}
{"severity":"INFO","time":"2021-12-24T09:28:09.111Z","message":"Running in ruby 2.7.5p203 (2021-11-24 revision f69aeb8314) [x86_64-linux]"}
{"severity":"INFO","time":"2021-12-24T09:28:09.111Z","message":"See LICENSE and the LGPL-3.0 for licensing details."}
{"severity":"INFO","time":"2021-12-24T09:28:09.111Z","message":"Upgrade to Sidekiq Pro for more features and support: https://sidekiq.org"}
{"severity":"INFO","time":"2021-12-24T09:28:10.225Z","message":"A worker terminated, shutting down the cluster"}
{"severity":"INFO","time":"2021-12-24T09:28:11.345Z","message":"Starting cluster with 1 processes"}

(This keeps repeating)

I found the issue after attaching a debugger to one of the sidekiq workers. It turns out they are trying to bind on port 8082/tcp by default which did not succeed on our pods due to unrelated configuration changes. Merry Christmas :slight_smile:

3 Likes

thanks for the information, I had the exact same issue and could fix it.
weird thing is that I was running for months with gitlab <14.6 and its web UI being served through port 8082.

We had this same issue, but in our case the Puma workers were running out of memory. Increasing the puma['per_worker_max_memory_mb'] setting helped stabilize things for us.

I had similar error message with sidekiq and constant reboot, in my case the metric exporter for sidekiq was turned off. So even though the port was already used by another service, it had no impact in this case.

sidekiq['metrics_enabled'] = false

I tried pretty much anything on this forum but ended up rolling back to previous 13.X version