Weird issues with build pipeline after upgrading to 14.6.0-ee

Hi everyone,

I am wondering if anyone else experienced weird issues after upgrading to 14.6.0. Since then our build pipeline has all sorts of strange issues which I will try to describe in this post.

When we create a new MR based on a branch that was fine before this happens:

As you can see “Checking pipeline status” spins there forever (I can confirm there is nothing wrong with gitlab-ci.yml whatsoever). This happens with any MR on any branch in any of our projects now.

Second issue is that pending pipelines are stuck in a weird state like this:

The pending (orange) state should actually say “failed”. Instead it keeps being like this forever. You cannot restart the pipeline, but you can restart a single job (which does not change anything there).

Another problem I see is when I try to create a new project from a template (no matter which one). It keeps doing this forever and the bundle process goes full power until I delete the project.

Screenshot 2021-12-23 at 13.37.46

I am running GitLab on a self-hosted Linux machine and would describe myself as an experienced DevOps which is why I would eliminate typical issues like “low disk space”. Unfortunately I have not been able to find anything useful in the logs.

I even tried a full re-install with omnibus and restoring one of our backups - problems persist. Is there something else I could try? I am very confused by the issues I see because they just don’t make sense.

Thanks :slight_smile:
Amadeus

I digged deeper into the logs and it seems sidekiq has some problems (see log attached).

What could be the reason for this / how can I debug this further?

{"severity":"INFO","time":"2021-12-24T09:27:19.092Z","message":"A worker terminated, shutting down the cluster"}
{"severity":"INFO","time":"2021-12-24T09:27:20.165Z","message":"Starting cluster with 1 processes"}
2021-12-24T09:27:35.221Z pid=16459 tid=5mj INFO: Listening on queues ["adjourned_project_deletion", "admin_emails", "analytics_code_review_metrics", "analytics_cycle_analytics_group_data_loader", "analytics_devops_adoption_create_snapshot", "analytics_usage_trends_counter_job", "app_sec_dast_scans_consistency", "approval_rules_external_approval_rule_payload", "approve_blocked_pending_approval_users", "audit_events_audit_event_streaming", "authorized_keys", "authorized_project_update:authorized_project_update_project_create", "authorized_project_update:authorized_project_update_project_group_link_create", "authorized_project_update:authorized_project_update_project_recalculate", "authorized_project_update:authorized_project_update_project_recalculate_per_user", "authorized_project_update:authorized_project_update_user_refresh_from_replica", "authorized_project_update:authorized_project_update_user_refresh_over_user_range", "authorized_project_update:authorized_project_update_user_refresh_with_low_urgency", "authorized_projects", "auto_devops:auto_devops_disable", "auto_merge:auto_merge_process", "auto_merge:merge_trains_refresh", "background_migration", "bulk_import", "bulk_imports_entity", "bulk_imports_export_request", "bulk_imports_pipeline", "bulk_imports_relation_export", "chaos:chaos_cpu_spin", "chaos:chaos_db_spin", "chaos:chaos_kill", "chaos:chaos_leak_mem", "chaos:chaos_sleep", "chat_notification", "ci_batch_reset_minutes", "ci_delete_objects", "ci_upstream_projects_subscriptions_cleanup", "container_repository:cleanup_container_repository", "container_repository:container_expiration_policies_cleanup_container_repository", "container_repository:delete_container_repository", "create_commit_signature", "create_github_webhook", "create_note_diff_file", "cronjob:active_user_count_threshold", "cronjob:adjourned_group_deletion", "cronjob:adjourned_projects_deletion_cron", "cronjob:admin_email", "cronjob:analytics_devops_adoption_create_all_snapshots", "cronjob:analytics_usage_trends_count_job_trigger", "cronjob:app_sec_dast_profile_schedule", "cronjob:authorized_project_update_periodic_recalculate", "cronjob:ci_archive_traces_cron", "cronjob:ci_delete_unit_tests", "cronjob:ci_pipeline_artifacts_expire_artifacts", "cronjob:ci_platform_metrics_update_cron", "cronjob:ci_schedule_delete_objects_cron", "cronjob:ci_stuck_builds_drop_running", "cronjob:ci_stuck_builds_drop_scheduled", "cronjob:clear_shared_runners_minutes", "cronjob:clusters_integrations_check_prometheus_health", "cronjob:container_expiration_policy", "cronjob:database_batched_background_migration", "cronjob:database_drop_detached_partitions", "cronjob:database_partition_management", "cronjob:dependency_proxy_image_ttl_group_policy", "cronjob:elastic_cluster_reindexing_cron", "cronjob:elastic_index_bulk_cron", "cronjob:elastic_index_initial_bulk_cron", "cronjob:elastic_migration", "cronjob:elastic_remove_expired_namespace_subscriptions_from_index_cron", "cronjob:environments_auto_delete_cron", "cronjob:environments_auto_stop_cron", "cronjob:expire_build_artifacts", "cronjob:geo_container_repository_sync_dispatch", "cronjob:geo_file_download_dispatch", "cronjob:geo_metrics_update", "cronjob:geo_prune_event_log", "cronjob:geo_registry_sync", "cronjob:geo_repository_sync", "cronjob:geo_repository_verification_primary_batch", "cronjob:geo_repository_verification_secondary_scheduler", "cronjob:geo_repository_verification_secondary_shard", "cronjob:geo_scheduler_per_shard_scheduler", "cronjob:geo_scheduler_primary_per_shard_scheduler", "cronjob:geo_scheduler_secondary_per_shard_scheduler", "cronjob:geo_secondary_registry_consistency", "cronjob:geo_secondary_usage_data_cron", "cronjob:geo_sidekiq_cron_config", "cronjob:geo_sync_timeout_cron", "cronjob:geo_verification_cron", "cronjob:gitlab_service_ping", "cronjob:historical_data", "cronjob:import_export_project_cleanup", "cronjob:import_software_licenses", "cronjob:import_stuck_project_import_jobs", "cronjob:incident_management_incident_sla_exceeded_check", "cronjob:incident_management_oncall_rotations_persist_all_rotations_shifts_job", "cronjob:incident_management_pending_escalations_schedule_check_cron", "cronjob:issue_due_scheduler", "cronjob:issues_reschedule_stuck_issue_rebalances", "cronjob:iterations_cadences_create_iterations", "cronjob:iterations_cadences_schedule_create_iterations", "cronjob:iterations_update_status", "cronjob:jira_import_stuck_jira_import_jobs", "cronjob:ldap_all_groups_sync", "cronjob:ldap_sync", "cronjob:loose_foreign_keys_cleanup", "cronjob:member_invitation_reminder_emails", "cronjob:metrics_dashboard_schedule_annotations_prune", "cronjob:namespaces_in_product_marketing_emails", "cronjob:namespaces_prune_aggregation_schedules", "cronjob:network_policy_metrics", "cronjob:packages_composer_cache_cleanup", "cronjob:pages_domain_removal_cron", "cronjob:pages_domain_ssl_renewal_cron", "cronjob:pages_domain_verification_cron", "cronjob:partition_creation", "cronjob:personal_access_tokens_expired_notification", "cronjob:personal_access_tokens_expiring", "cronjob:pipeline_schedule", "cronjob:prune_old_events", "cronjob:pseudonymizer", "cronjob:releases_manage_evidence", "cronjob:remove_expired_group_links", "cronjob:remove_expired_members", "cronjob:remove_unaccepted_member_invites", "cronjob:remove_unreferenced_lfs_objects", "cronjob:repository_archive_cache", "cronjob:repository_check_dispatch", "cronjob:requests_profiles", "cronjob:schedule_merge_request_cleanup_refs", "cronjob:schedule_migrate_external_diffs", "cronjob:security_create_orchestration_policy", "cronjob:security_orchestration_policy_rule_schedule", "cronjob:ssh_keys_expired_notification", "cronjob:ssh_keys_expiring_soon_notification", "cronjob:stuck_ci_jobs", "cronjob:stuck_export_jobs", "cronjob:stuck_merge_jobs", "cronjob:sync_seat_link", "cronjob:trending_projects", "cronjob:update_all_mirrors", "cronjob:update_container_registry_info", "cronjob:update_max_seats_used_for_gitlab_com_subscriptions", "cronjob:user_status_cleanup_batch", "cronjob:users_create_statistics", "cronjob:users_deactivate_dormant_users", "cronjob:vulnerabilities_historical_statistics_deletion", "cronjob:vulnerabilities_statistics_schedule", "cronjob:x509_issuer_crl_check", "dast_site_validation", "default", "delete_diff_files", "delete_merged_branches", "delete_stored_files", "delete_user", "dependency_proxy:purge_dependency_proxy_cache", "dependency_proxy_blob:dependency_proxy_cleanup_blob", "dependency_proxy_manifest:dependency_proxy_cleanup_manifest", "deployment:deployments_archive_in_project", "deployment:deployments_auto_rollback", "deployment:deployments_drop_older_deployments", "deployment:deployments_hooks", "deployment:deployments_link_merge_request", "deployment:deployments_update_environment", "design_management_copy_design_collection", "design_management_new_version", "destroy_pages_deployments", "detect_repository_languages", "disallow_two_factor_for_group", "disallow_two_factor_for_subgroups", "dora_metrics:dora_daily_metrics_refresh", "elastic_association_indexer", "elastic_commit_indexer", "elastic_delete_project", "elastic_full_index", "elastic_indexing_control", "elastic_namespace_indexer", "elastic_namespace_rollout", "email_receiver", "emails_on_push", "environments_auto_stop", "environments_canary_ingress_update", "epics:epics_update_epics_dates", "epics_new_epic_issue", "error_tracking_issue_link", "experiments_record_conversion_event", "expire_build_instance_artifacts", "export_csv", "external_service_reactive_caching", "file_hook", "flush_counter_increments", "gcp_cluster:cluster_configure_istio", "gcp_cluster:cluster_install_app", "gcp_cluster:cluster_patch_app", "gcp_cluster:cluster_provision", "gcp_cluster:cluster_update_app", "gcp_cluster:cluster_upgrade_app", "gcp_cluster:cluster_wait_for_app_installation", "gcp_cluster:cluster_wait_for_app_update", "gcp_cluster:cluster_wait_for_ingress_ip_address", "gcp_cluster:clusters_applications_activate_service", "gcp_cluster:clusters_applications_deactivate_service", "gcp_cluster:clusters_applications_uninstall", "gcp_cluster:clusters_applications_wait_for_uninstall_app", "gcp_cluster:clusters_cleanup_project_namespace", "gcp_cluster:clusters_cleanup_service_account", "gcp_cluster:wait_for_cluster_creation", "geo:geo_batch_event_create", "geo:geo_batch_project_registry", "geo:geo_batch_project_registry_scheduler", "geo:geo_container_repository_sync", "geo:geo_design_repository_shard_sync", "geo:geo_design_repository_sync", "geo:geo_destroy", "geo:geo_event", "geo:geo_file_download", "geo:geo_file_registry_removal", "geo:geo_file_removal", "geo:geo_hashed_storage_attachments_migration", "geo:geo_hashed_storage_migration", "geo:geo_project_sync", "geo:geo_rename_repository", "geo:geo_repositories_clean_up", "geo:geo_repository_cleanup", "geo:geo_repository_destroy", "geo:geo_repository_shard_sync", "geo:geo_repository_verification_primary_shard", "geo:geo_repository_verification_primary_single", "geo:geo_repository_verification_secondary_single", "geo:geo_reverification_batch", "geo:geo_scheduler_primary_scheduler", "geo:geo_scheduler_scheduler", "geo:geo_scheduler_secondary_scheduler", "geo:geo_verification", "geo:geo_verification_batch", "geo:geo_verification_state_backfill", "geo:geo_verification_timeout", "github_import_advance_stage", "github_importer:github_import_import_diff_note", "github_importer:github_import_import_issue", "github_importer:github_import_import_lfs_object", "github_importer:github_import_import_note", "github_importer:github_import_import_pull_request", "github_importer:github_import_import_pull_request_merged_by", "github_importer:github_import_import_pull_request_review", "github_importer:github_import_refresh_import_jid", "github_importer:github_import_stage_finish_import", "github_importer:github_import_stage_import_base_data", "github_importer:github_import_stage_import_issues_and_diff_notes", "github_importer:github_import_stage_import_lfs_objects", "github_importer:github_import_stage_import_notes", "github_importer:github_import_stage_import_pull_requests", "github_importer:github_import_stage_import_pull_requests_merged_by", "github_importer:github_import_stage_import_pull_requests_reviews", "github_importer:github_import_stage_import_repository", "gitlab_performance_bar_stats", "gitlab_shell", "group_destroy", "group_export", "group_import", "group_saml_group_sync", "group_wikis_git_garbage_collect", "groups_create_event", "groups_export_memberships", "groups_schedule_bulk_repository_shard_moves", "groups_update_repository_storage", "groups_update_statistics", "hashed_storage:hashed_storage_migrator", "hashed_storage:hashed_storage_project_migrate", "hashed_storage:hashed_storage_project_rollback", "hashed_storage:hashed_storage_rollbacker", "import_issues_csv", "incident_management:incident_management_add_severity_system_note", "incident_management:incident_management_pager_duty_process_incident", "incident_management:incident_management_process_alert_worker_v2", "incident_management_apply_incident_sla_exceeded_label", "incident_management_oncall_rotations_persist_shifts_job", "incident_management_pending_escalations_alert_check", "incident_management_pending_escalations_alert_create", "incident_management_pending_escalations_issue_check", "integrations_create_external_cross_reference", "invalid_gpg_signature_update", "irker", "issuable_export_csv", "issuable_label_links_destroy", "issuables_clear_groups_issue_counter", "issue_placement", "issue_rebalancing", "issues_placement", "issues_rebalancing", "iterations:iterations_roll_over_issues", "jira_connect:jira_connect_forward_event", "jira_connect:jira_connect_retry_request", "jira_connect:jira_connect_sync_branch", "jira_connect:jira_connect_sync_builds", "jira_connect:jira_connect_sync_deployments", "jira_connect:jira_connect_sync_feature_flags", "jira_connect:jira_connect_sync_merge_request", "jira_connect:jira_connect_sync_project", "jira_importer:jira_import_advance_stage", "jira_importer:jira_import_import_issue", "jira_importer:jira_import_stage_finish_import", "jira_importer:jira_import_stage_import_attachments", "jira_importer:jira_import_stage_import_issues", "jira_importer:jira_import_stage_import_labels", "jira_importer:jira_import_stage_import_notes", "jira_importer:jira_import_stage_start_import", "ldap_group_sync", "mail_scheduler:mail_scheduler_issue_due", "mail_scheduler:mail_scheduler_notification_service", "mailers", "merge", "merge_request_cleanup_refs", "merge_request_mergeability_check", "merge_request_reset_approvals", "merge_requests_delete_source_branch", "merge_requests_handle_assignees_change", "merge_requests_resolve_todos", "merge_requests_sync_code_owner_approval_rules", "metrics_dashboard_prune_old_annotations", "metrics_dashboard_sync_dashboards", "migrate_external_diffs", "namespaceless_project_destroy", "namespaces_invite_team_email", "namespaces_onboarding_issue_created", "namespaces_onboarding_pipeline_created", "namespaces_onboarding_progress", "namespaces_onboarding_user_added", "namespaces_process_sync_events", "namespaces_sync_namespace_name", "new_epic", "new_issue", "new_merge_request", "new_note", "object_pool:object_pool_create", "object_pool:object_pool_destroy", "object_pool:object_pool_join", "object_pool:object_pool_schedule_join", "object_storage:object_storage_background_move", "object_storage:object_storage_migrate_uploads", "package_repositories:packages_debian_generate_distribution", "package_repositories:packages_debian_process_changes", "package_repositories:packages_go_sync_packages", "package_repositories:packages_helm_extraction", "package_repositories:packages_maven_metadata_sync", "package_repositories:packages_nuget_extraction", "package_repositories:packages_rubygems_extraction", "packages_composer_cache_update", "pages", "pages_domain_ssl_renewal", "pages_domain_verification", "pages_transfer", "pages_update_configuration", "personal_access_tokens:personal_access_tokens_groups_policy", "personal_access_tokens:personal_access_tokens_instance_policy", "phabricator_import_import_tasks", "pipeline_background:archive_trace", "pipeline_background:ci_archive_trace", "pipeline_background:ci_build_trace_chunk_flush", "pipeline_background:ci_daily_build_group_report_results", "pipeline_background:ci_minutes_update_project_and_namespace_usage", "pipeline_background:ci_pending_builds_update_group", "pipeline_background:ci_pending_builds_update_project", "pipeline_background:ci_pipeline_artifacts_coverage_report", "pipeline_background:ci_pipeline_artifacts_create_quality_report", "pipeline_background:ci_pipeline_success_unlock_artifacts", "pipeline_background:ci_ref_delete_unlock_artifacts", "pipeline_background:ci_sync_reports_to_report_approval_rules", "pipeline_background:ci_test_failure_history", "pipeline_cache:expire_job_cache", "pipeline_cache:expire_pipeline_cache", "pipeline_creation:ci_external_pull_requests_create_pipeline", "pipeline_creation:create_pipeline", "pipeline_creation:merge_requests_create_pipeline", "pipeline_creation:run_pipeline_schedule", "pipeline_default:ci_create_cross_project_pipeline", "pipeline_default:ci_create_downstream_pipeline", "pipeline_default:ci_drop_pipeline", "pipeline_default:ci_merge_requests_add_todo_when_build_fails", "pipeline_default:ci_pipeline_bridge_status", "pipeline_default:ci_retry_pipeline", "pipeline_default:ci_trigger_downstream_subscriptions", "pipeline_default:pipeline_metrics", "pipeline_default:pipeline_notification", "pipeline_hooks:build_hooks", "pipeline_hooks:pipeline_hooks", "pipeline_processing:build_finished", "pipeline_processing:build_queue", "pipeline_processing:build_success", "pipeline_processing:ci_build_finished", "pipeline_processing:ci_build_prepare", "pipeline_processing:ci_build_schedule", "pipeline_processing:ci_initial_pipeline_process", "pipeline_processing:ci_resource_groups_assign_resource_from_resource_group", "pipeline_processing:pipeline_process", "pipeline_processing:stage_update", "pipeline_processing:update_head_pipeline_for_merge_request", "post_receive", "process_commit", "project_cache", "project_daily_statistics", "project_destroy", "project_export", "project_import_schedule", "project_service", "project_template_export", "projects_git_garbage_collect", "projects_post_creation", "projects_process_sync_events", "projects_schedule_bulk_repository_shard_moves", "projects_update_repository_storage", "prometheus_create_default_alerts", "propagate_integration", "propagate_integration_group", "propagate_integration_inherit", "propagate_integration_inherit_descendant", "propagate_integration_project", "reactive_caching", "rebase", "refresh_license_compliance_checks", "releases_create_evidence", "remote_mirror_notification", "repository_check:repository_check_batch", "repository_check:repository_check_clear", "repository_check:repository_check_single_repository", "repository_cleanup", "repository_fork", "repository_import", "repository_push_audit_event", "repository_remove_remote", "repository_update_mirror", "repository_update_remote_mirror", "requirements_management_import_requirements_csv", "requirements_management_process_requirements_reports", "security_auto_fix", "security_scans:scan_security_report_secrets", "security_scans:security_store_scans", "security_scans:security_track_secure_scans", "security_scans:store_security_reports", "self_monitoring_project_create", "self_monitoring_project_delete", "service_desk_email_receiver", "set_user_status_based_on_user_cap_setting", "snippets_schedule_bulk_repository_shard_moves", "snippets_update_repository_storage", "status_page_publish", "sync_seat_link_request", "system_hook_push", "tasks_to_be_done_create", "todos_destroyer:todos_destroyer_confidential_epic", "todos_destroyer:todos_destroyer_confidential_issue", "todos_destroyer:todos_destroyer_destroyed_designs", "todos_destroyer:todos_destroyer_destroyed_issuable", "todos_destroyer:todos_destroyer_entity_leave", "todos_destroyer:todos_destroyer_group_private", "todos_destroyer:todos_destroyer_private_features", "todos_destroyer:todos_destroyer_project_private", "unassign_issuables:members_destroyer_unassign_issuables", "update_external_pull_requests", "update_highest_role", "update_merge_requests", "update_namespace_statistics:namespaces_root_statistics", "update_namespace_statistics:namespaces_schedule_aggregation", "update_project_statistics", "upload_checksum", "vulnerabilities_statistics_adjustment", "vulnerability_exports_export", "vulnerability_exports_export_deletion", "web_hook", "web_hooks_destroy", "web_hooks_log_execution", "wikis_git_garbage_collect", "x509_certificate_revoke"]
{"severity":"INFO","time":"2021-12-24T09:27:35.224Z","message":"Booting Sidekiq 6.3.1 with redis options {:instrumentation_class=\u003eGitlab::Instrumentation::Redis::Queues, :path=\u003e\"/var/opt/gitlab/redis/redis.socket\", :namespace=\u003e\"resque:gitlab\"}"}
{"severity":"INFO","time":"2021-12-24T09:27:38.096Z","message":"GitLab reliable fetch activated!"}
{"severity":"INFO","time":"2021-12-24T09:27:38.226Z","message":"Cron Jobs - deleted job with name: update_all_mirrors_worker"}
{"severity":"INFO","time":"2021-12-24T09:27:38.228Z","message":"Cron Jobs - deleted job with name: geo_sidekiq_cron_config_worker"}
{"severity":"INFO","time":"2021-12-24T09:27:38.230Z","message":"Cron Jobs - add job with name: stuck_ci_jobs_worker"}
{"severity":"INFO","time":"2021-12-24T09:27:38.232Z","message":"Cron Jobs - add job with name: pipeline_schedule_worker"}
[...]
{"severity":"WARN","time":"2021-12-24T09:27:40.052Z","message":"Creating scope :without_statuses. Overwriting existing method CommitStatus.without_statuses."}
{"severity":"WARN","time":"2021-12-24T09:27:40.102Z","message":"Creating scope :without_statuses. Overwriting existing method Ci::Pipeline.without_statuses."}
{"severity":"WARN","time":"2021-12-24T09:27:40.302Z","message":"Creating scope :in_pipelines. Overwriting existing method Ci::Build.in_pipelines."}
{"severity":"WARN","time":"2021-12-24T09:27:40.945Z","message":"Creating scope :of_projects. Overwriting existing method MergeRequest.of_projects."}
{"severity":"WARN","time":"2021-12-24T09:27:40.946Z","message":"Creating scope :join_project. Overwriting existing method MergeRequest.join_project."}
{"severity":"WARN","time":"2021-12-24T09:27:40.947Z","message":"Creating scope :references_project. Overwriting existing method MergeRequest.references_project."}
{"severity":"WARN","time":"2021-12-24T09:27:40.986Z","message":"Creating scope :system. Overwriting existing method Note.system."}
{"severity":"WARN","time":"2021-12-24T09:27:41.348Z","message":"Creating scope :closed. Overwriting existing method Iteration.closed."}
{"severity":"WARN","time":"2021-12-24T09:27:41.348Z","message":"Creating scope :within_timeframe. Overwriting existing method Iteration.within_timeframe."}
{"severity":"WARN","time":"2021-12-24T09:27:41.720Z","message":"Creating scope :opened. Overwriting existing method Epic.opened."}
{"severity":"WARN","time":"2021-12-24T09:27:41.720Z","message":"Creating scope :closed. Overwriting existing method Epic.closed."}
{"severity":"WARN","time":"2021-12-24T09:27:41.922Z","message":"Creating scope :order_created_desc. Overwriting existing method Packages::Package.order_created_desc."}
{"severity":"WARN","time":"2021-12-24T09:27:41.922Z","message":"Creating scope :order_name_desc. Overwriting existing method Packages::Package.order_name_desc."}
{"severity":"WARN","time":"2021-12-24T09:27:43.781Z","message":"Creating scope :open. Overwriting existing method AlertManagement::Alert.open."}
{"severity":"WARN","time":"2021-12-24T09:27:43.857Z","message":"Creating scope :with_associations. Overwriting existing method Board.with_associations."}
{"severity":"WARN","time":"2021-12-24T09:27:43.929Z","message":"Creating scope :movable. Overwriting existing method Boards::EpicList.movable."}
{"severity":"WARN","time":"2021-12-24T09:27:44.292Z","message":"Creating scope :order_name_asc. Overwriting existing method Label.order_name_asc."}
{"severity":"WARN","time":"2021-12-24T09:27:44.293Z","message":"Creating scope :order_name_desc. Overwriting existing method Label.order_name_desc."}
{"severity":"WARN","time":"2021-12-24T09:27:47.207Z","message":"Creating scope :with_logging_info. Overwriting existing method PagesDomain.with_logging_info."}
{"severity":"WARN","time":"2021-12-24T09:27:47.236Z","message":"Creating scope :system. Overwriting existing method PrometheusMetric.system."}
{"severity":"WARN","time":"2021-12-24T09:27:47.432Z","message":"Creating scope :open. Overwriting existing method ExternalPullRequest.open."}
{"severity":"WARN","time":"2021-12-24T09:27:47.498Z","message":"Creating scope :order_name_asc. Overwriting existing method Member.order_name_asc."}
{"severity":"WARN","time":"2021-12-24T09:27:47.499Z","message":"Creating scope :order_name_desc. Overwriting existing method Member.order_name_desc."}
{"severity":"WARN","time":"2021-12-24T09:27:47.678Z","message":"Creating scope :with_user. Overwriting existing method GroupMember.with_user."}
{"severity":"WARN","time":"2021-12-24T09:27:47.680Z","message":"Creating scope :reporters. Overwriting existing method GroupMember.reporters."}
{"severity":"WARN","time":"2021-12-24T09:27:47.680Z","message":"Creating scope :guests. Overwriting existing method GroupMember.guests."}
{"severity":"WARN","time":"2021-12-24T09:27:47.943Z","message":"Creating scope :extended. Overwriting existing method GitlabSubscription.extended."}
{"severity":"WARN","time":"2021-12-24T09:27:48.370Z","message":"Creating scope :without_statuses. Overwriting existing method Ci::Stage.without_statuses."}
{"severity":"WARN","time":"2021-12-24T09:27:48.370Z","message":"Creating scope :created. Overwriting existing method Ci::Stage.created."}
{"severity":"WARN","time":"2021-12-24T09:27:48.371Z","message":"Creating scope :pending. Overwriting existing method Ci::Stage.pending."}
{"severity":"WARN","time":"2021-12-24T09:27:48.371Z","message":"Creating scope :running. Overwriting existing method Ci::Stage.running."}
{"severity":"WARN","time":"2021-12-24T09:27:48.371Z","message":"Creating scope :success. Overwriting existing method Ci::Stage.success."}
{"severity":"WARN","time":"2021-12-24T09:27:48.372Z","message":"Creating scope :failed. Overwriting existing method Ci::Stage.failed."}
{"severity":"WARN","time":"2021-12-24T09:27:48.372Z","message":"Creating scope :canceled. Overwriting existing method Ci::Stage.canceled."}
{"severity":"WARN","time":"2021-12-24T09:27:48.373Z","message":"Creating scope :skipped. Overwriting existing method Ci::Stage.skipped."}
{"severity":"WARN","time":"2021-12-24T09:27:48.373Z","message":"Creating scope :manual. Overwriting existing method Ci::Stage.manual."}
{"severity":"WARN","time":"2021-12-24T09:27:48.373Z","message":"Creating scope :scheduled. Overwriting existing method Ci::Stage.scheduled."}
{"severity":"WARN","time":"2021-12-24T09:27:48.374Z","message":"Creating scope :preparing. Overwriting existing method Ci::Stage.preparing."}
{"severity":"WARN","time":"2021-12-24T09:27:48.374Z","message":"Creating scope :waiting_for_resource. Overwriting existing method Ci::Stage.waiting_for_resource."}
{"severity":"WARN","time":"2021-12-24T09:27:48.442Z","message":"Creating scope :database. Overwriting existing method Ci::BuildTraceChunk.database."}
{"severity":"WARN","time":"2021-12-24T09:27:49.555Z","message":"Creating scope :approved_by_committer. Overwriting existing method MergeRequests::ComplianceViolation.approved_by_committer."}
{"severity":"WARN","time":"2021-12-24T09:27:49.559Z","message":"Creating scope :runnable_schedules. Overwriting existing method Security::OrchestrationPolicyRuleSchedule.runnable_schedules."}
{"severity":"WARN","time":"2021-12-24T09:27:50.676Z","message":"Creating scope :regular. Overwriting existing method EE::Gitlab::BackgroundMigration::MigrateApproverToApprovalRules::ApprovalMergeRequestRule.regular."}
{"severity":"WARN","time":"2021-12-24T09:27:50.677Z","message":"Creating scope :code_owner. Overwriting existing method EE::Gitlab::BackgroundMigration::MigrateApproverToApprovalRules::ApprovalMergeRequestRule.code_owner."}
{"severity":"WARN","time":"2021-12-24T09:27:53.223Z","message":"Creating scope :system. Overwriting existing method Gitlab::DatabaseImporters::CommonMetrics::PrometheusMetric.system."}
{"severity":"INFO","time":"2021-12-24T09:28:09.111Z","message":"Booted Rails 6.1.4.1 application in production environment"}
{"severity":"INFO","time":"2021-12-24T09:28:09.111Z","message":"Running in ruby 2.7.5p203 (2021-11-24 revision f69aeb8314) [x86_64-linux]"}
{"severity":"INFO","time":"2021-12-24T09:28:09.111Z","message":"See LICENSE and the LGPL-3.0 for licensing details."}
{"severity":"INFO","time":"2021-12-24T09:28:09.111Z","message":"Upgrade to Sidekiq Pro for more features and support: https://sidekiq.org"}
{"severity":"INFO","time":"2021-12-24T09:28:10.225Z","message":"A worker terminated, shutting down the cluster"}
{"severity":"INFO","time":"2021-12-24T09:28:11.345Z","message":"Starting cluster with 1 processes"}

(This keeps repeating)

I found the issue after attaching a debugger to one of the sidekiq workers. It turns out they are trying to bind on port 8082/tcp by default which did not succeed on our pods due to unrelated configuration changes. Merry Christmas :slight_smile:

5 Likes

thanks for the information, I had the exact same issue and could fix it.
weird thing is that I was running for months with gitlab <14.6 and its web UI being served through port 8082.

We had this same issue, but in our case the Puma workers were running out of memory. Increasing the puma['per_worker_max_memory_mb'] setting helped stabilize things for us.