After upgrading GitLab to 14.7, sidekiq fails to stay running, and no jobs are running.
All CI jobs are stuck saying This job depends on upstream jobs that need to succeed in order for this job to be triggered.
Kubernetes version: 1.21.6
Helm chart version: 5.7.0 (gitlab 14.7.0)
Output of kubectl get pods
NAME READY STATUS RESTARTS AGE
db-postgresql-0 1/1 Running 0 19h
gitlab-gitaly-0 1/1 Running 0 6h8m
gitlab-gitlab-exporter-74669c96b7-d7wpz 1/1 Running 0 6h9m
gitlab-gitlab-pages-657d7f6ffc-d9vlt 1/1 Running 0 6h9m
gitlab-gitlab-runner-75b77988c8-g2q88 1/1 Running 5 6h9m
gitlab-gitlab-shell-5d64558c69-gzjsz 1/1 Running 0 6h9m
gitlab-gitlab-shell-5d64558c69-j45qg 1/1 Running 0 6h3m
gitlab-gitlab-upgrade-check-gfjql 0/1 Completed 0 3h53m
gitlab-migrations-14-x8g8m 0/1 Completed 4 4h15m
gitlab-minio-864888b9fb-2rmmn 1/1 Running 7 6h9m
gitlab-minio-create-buckets-14-fqkbc 0/1 Completed 0 4h15m
gitlab-prometheus-server-5b47c879b4-p7k4h 2/2 Running 0 19h
gitlab-registry-b6f64686b-jrn8v 1/1 Running 6 6h9m
gitlab-shared-secrets-13-jl6-l8cbj 0/1 Completed 0 5h7m
gitlab-shared-secrets-15-awv-mtqzf 0/1 Completed 0 3h53m
gitlab-sidekiq-all-in-1-v2-ff654dcd9-n88fh 0/1 Running 12 55m
gitlab-toolbox-7bd697cb59-nrpcv 1/1 Running 0 6h9m
gitlab-webservice-default-6f67f58788-dmvnw 2/2 Running 8 6h9m
gitlab-webservice-default-6f67f58788-tfwh8 2/2 Running 16 5h42m
redis-master-0 1/1 Running 1 19h
Last bunch of lines from the sidekiq container
Creating scope :order_name_asc. Overwriting existing method Label.order_name_asc.
Creating scope :order_name_desc. Overwriting existing method Label.order_name_desc.
2022-01-28T21:35:42.947Z pid=22 tid=9k6 WARN: Creating scope :open. Overwriting existing method ExternalPullRequest.open.
Creating scope :open. Overwriting existing method ExternalPullRequest.open.
2022-01-28T21:35:43.563Z pid=22 tid=9k6 WARN: Creating scope :order_name_asc. Overwriting existing method Member.order_name_asc.
2022-01-28T21:35:43.564Z pid=22 tid=9k6 WARN: Creating scope :order_name_desc. Overwriting existing method Member.order_name_desc.
2022-01-28T21:35:43.914Z pid=22 tid=9k6 WARN: Creating scope :with_logging_info. Overwriting existing method PagesDomain.with_logging_info.
Creating scope :order_name_asc. Overwriting existing method Member.order_name_asc.
Creating scope :order_name_desc. Overwriting existing method Member.order_name_desc.
Creating scope :with_logging_info. Overwriting existing method PagesDomain.with_logging_info.
2022-01-28T21:35:44.564Z pid=22 tid=9k6 WARN: Creating scope :system. Overwriting existing method PrometheusMetric.system.
Creating scope :system. Overwriting existing method PrometheusMetric.system.
2022-01-28T21:35:45.568Z pid=22 tid=9k6 WARN: Creating scope :with_user. Overwriting existing method GroupMember.with_user.
2022-01-28T21:35:45.579Z pid=22 tid=9k6 WARN: Creating scope :reporters. Overwriting existing method GroupMember.reporters.
2022-01-28T21:35:45.580Z pid=22 tid=9k6 WARN: Creating scope :guests. Overwriting existing method GroupMember.guests.
Creating scope :with_user. Overwriting existing method GroupMember.with_user.
Creating scope :reporters. Overwriting existing method GroupMember.reporters.
Creating scope :guests. Overwriting existing method GroupMember.guests.
2022-01-28T21:35:47.612Z pid=22 tid=9k6 WARN: Creating scope :extended. Overwriting existing method GitlabSubscription.extended.
Creating scope :extended. Overwriting existing method GitlabSubscription.extended.
2022-01-28T21:35:51.276Z pid=22 tid=9k6 WARN: Creating scope :without_statuses. Overwriting existing method Ci::Stage.without_statuses.
2022-01-28T21:35:51.279Z pid=22 tid=9k6 WARN: Creating scope :created. Overwriting existing method Ci::Stage.created.
2022-01-28T21:35:51.280Z pid=22 tid=9k6 WARN: Creating scope :pending. Overwriting existing method Ci::Stage.pending.
2022-01-28T21:35:51.281Z pid=22 tid=9k6 WARN: Creating scope :running. Overwriting existing method Ci::Stage.running.
2022-01-28T21:35:51.282Z pid=22 tid=9k6 WARN: Creating scope :success. Overwriting existing method Ci::Stage.success.
2022-01-28T21:35:51.283Z pid=22 tid=9k6 WARN: Creating scope :failed. Overwriting existing method Ci::Stage.failed.
2022-01-28T21:35:51.284Z pid=22 tid=9k6 WARN: Creating scope :canceled. Overwriting existing method Ci::Stage.canceled.
2022-01-28T21:35:51.284Z pid=22 tid=9k6 WARN: Creating scope :skipped. Overwriting existing method Ci::Stage.skipped.
2022-01-28T21:35:51.285Z pid=22 tid=9k6 WARN: Creating scope :manual. Overwriting existing method Ci::Stage.manual.
2022-01-28T21:35:51.293Z pid=22 tid=9k6 WARN: Creating scope :scheduled. Overwriting existing method Ci::Stage.scheduled.
2022-01-28T21:35:51.294Z pid=22 tid=9k6 WARN: Creating scope :preparing. Overwriting existing method Ci::Stage.preparing.
2022-01-28T21:35:51.296Z pid=22 tid=9k6 WARN: Creating scope :waiting_for_resource. Overwriting existing method Ci::Stage.waiting_for_resource.
2022-01-28T21:35:51.680Z pid=22 tid=9k6 WARN: Creating scope :database. Overwriting existing method Ci::BuildTraceChunk.database.
Creating scope :without_statuses. Overwriting existing method Ci::Stage.without_statuses.
Creating scope :created. Overwriting existing method Ci::Stage.created.
Creating scope :pending. Overwriting existing method Ci::Stage.pending.
Creating scope :running. Overwriting existing method Ci::Stage.running.
Creating scope :success. Overwriting existing method Ci::Stage.success.
Creating scope :failed. Overwriting existing method Ci::Stage.failed.
Creating scope :canceled. Overwriting existing method Ci::Stage.canceled.
Creating scope :skipped. Overwriting existing method Ci::Stage.skipped.
Creating scope :manual. Overwriting existing method Ci::Stage.manual.
Creating scope :scheduled. Overwriting existing method Ci::Stage.scheduled.
Creating scope :preparing. Overwriting existing method Ci::Stage.preparing.
Creating scope :waiting_for_resource. Overwriting existing method Ci::Stage.waiting_for_resource.
Creating scope :database. Overwriting existing method Ci::BuildTraceChunk.database.
2022-01-28T21:35:59.333Z pid=22 tid=9k6 WARN: Creating scope :approved_by_committer. Overwriting existing method MergeRequests::ComplianceViolation.approved_by_committer.
2022-01-28T21:35:59.508Z pid=22 tid=9k6 WARN: Creating scope :runnable_schedules. Overwriting existing method Security::OrchestrationPolicyRuleSchedule.runnable_schedules.
Creating scope :approved_by_committer. Overwriting existing method MergeRequests::ComplianceViolation.approved_by_committer.
Creating scope :runnable_schedules. Overwriting existing method Security::OrchestrationPolicyRuleSchedule.runnable_schedules.
2022-01-28T21:36:06.992Z pid=22 tid=9k6 WARN: Creating scope :regular. Overwriting existing method EE::Gitlab::BackgroundMigration::MigrateApproverToApprovalRules::ApprovalMergeRequestRule.regular.
2022-01-28T21:36:06.994Z pid=22 tid=9k6 WARN: Creating scope :code_owner. Overwriting existing method EE::Gitlab::BackgroundMigration::MigrateApproverToApprovalRules::ApprovalMergeRequestRule.code_owner.
Creating scope :regular. Overwriting existing method EE::Gitlab::BackgroundMigration::MigrateApproverToApprovalRules::ApprovalMergeRequestRule.regular.
Creating scope :code_owner. Overwriting existing method EE::Gitlab::BackgroundMigration::MigrateApproverToApprovalRules::ApprovalMergeRequestRule.code_owner.
2022-01-28T21:36:21.933Z pid=22 tid=9k6 WARN: Creating scope :system. Overwriting existing method Gitlab::DatabaseImporters::CommonMetrics::PrometheusMetric.system.
Creating scope :system. Overwriting existing method Gitlab::DatabaseImporters::CommonMetrics::PrometheusMetric.system.
{"severity":"INFO","time":"2022-01-28T21:36:58.434Z","message":"A worker terminated, shutting down the cluster"}