Error 502 on upgrade from 14.6.7 > 14.7.0

I’ve been running GitLab (omnibus, Ubuntu 20.04, amd64) in version 14.6.7 for quite some time and now wanted to start an upgrade path to the most recent 15.X release.

First step was an upgrade to 14.7.0 but this is already causing problems. I downloaded the corresponding deb archive from https://packages.gitlab.com and used dpkg to install it. All seemed fine, no errors and GitLab also seemed to start up. No changes to the configuration file, the vast majorty of options (except for the usual one, directory paths, ldap etc) have been at their defaults since I installed the very first version of GitLab.

However, the 502 error didn’t disappear even after minutes.

gitlab-ctl status shows all processes running fine but the uptime of puma and sidekiq keeps dropping to a few seconds - they are both restarting continuously it seems.

I then ran gitlab-ctl tail puma:

==> /var/log/gitlab/puma/puma_stdout.log <==
{"timestamp":"2022-10-29T06:56:34.554Z","pid":47170,"message":"PumaWorkerKiller: Consuming 4277.15234375 mb with master and 4 workers."}
{"timestamp":"2022-10-29T06:56:54.556Z","pid":47170,"message":"PumaWorkerKiller: Consuming 4283.84375 mb with master and 4 workers."}
{"timestamp":"2022-10-29T06:57:14.559Z","pid":47170,"message":"PumaWorkerKiller: Consuming 4292.703125 mb with master and 4 workers."}
{"timestamp":"2022-10-29T06:57:34.561Z","pid":47170,"message":"PumaWorkerKiller: Consuming 4319.90625 mb with master and 4 workers."}
{"timestamp":"2022-10-29T06:57:54.562Z","pid":47170,"message":"PumaWorkerKiller: Consuming 4325.0546875 mb with master and 4 workers."}
{"timestamp":"2022-10-29T06:58:14.565Z","pid":47170,"message":"PumaWorkerKiller: Consuming 4325.0078125 mb with master and 4 workers."}
{"timestamp":"2022-10-29T06:58:34.566Z","pid":47170,"message":"PumaWorkerKiller: Consuming 4328.3671875 mb with master and 4 workers."}
{"timestamp":"2022-10-29T06:58:42.158Z","pid":47170,"message":"=== puma shutdown: 2022-10-29 08:58:42 +0200 ==="}
{"timestamp":"2022-10-29T06:58:42.158Z","pid":47170,"message":"- Goodbye!"}
{"timestamp":"2022-10-29T06:58:42.158Z","pid":47170,"message":"- Gracefully shutting down workers..."}

==> /var/log/gitlab/puma/current <==
2022-10-29_06:59:48.31015   /opt/gitlab/embedded/bin/puma:23:in `<top (required)>'
2022-10-29_06:59:49.09474 {"timestamp":"2022-10-29T06:59:49.094Z","pid":61021,"message":"Puma starting in cluster mode..."}
2022-10-29_06:59:49.09480 {"timestamp":"2022-10-29T06:59:49.094Z","pid":61021,"message":"* Puma version: 5.5.2 (ruby 2.7.5-p203) (\"Zawgyi\")"}
2022-10-29_06:59:49.09483 {"timestamp":"2022-10-29T06:59:49.094Z","pid":61021,"message":"*  Min threads: 4"}
2022-10-29_06:59:49.09484 {"timestamp":"2022-10-29T06:59:49.094Z","pid":61021,"message":"*  Max threads: 4"}
2022-10-29_06:59:49.09486 {"timestamp":"2022-10-29T06:59:49.094Z","pid":61021,"message":"*  Environment: production"}
2022-10-29_06:59:49.09489 {"timestamp":"2022-10-29T06:59:49.094Z","pid":61021,"message":"*   Master PID: 61021"}
2022-10-29_06:59:49.09491 {"timestamp":"2022-10-29T06:59:49.094Z","pid":61021,"message":"*      Workers: 4"}
2022-10-29_06:59:49.09493 {"timestamp":"2022-10-29T06:59:49.094Z","pid":61021,"message":"*     Restarts: (✔) hot (✖) phased"}
2022-10-29_06:59:49.09495 {"timestamp":"2022-10-29T06:59:49.094Z","pid":61021,"message":"* Preloading application"}
2022-10-29_06:59:58.34045 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab.rb:47: warning: already initialized constant Gitlab::APP_DIRS_PATTERN
2022-10-29_06:59:58.34059 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab.rb:47: warning: previous definition of APP_DIRS_PATTERN was here
2022-10-29_06:59:58.34067 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab.rb:48: warning: already initialized constant Gitlab::VERSION
2022-10-29_06:59:58.34070 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab.rb:48: warning: previous definition of VERSION was here
2022-10-29_06:59:58.34076 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab.rb:49: warning: already initialized constant Gitlab::INSTALLATION_TYPE
2022-10-29_06:59:58.34079 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab.rb:49: warning: previous definition of INSTALLATION_TYPE was here
2022-10-29_06:59:58.34082 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab.rb:50: warning: already initialized constant Gitlab::HTTP_PROXY_ENV_VARS
2022-10-29_06:59:58.34084 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab.rb:50: warning: previous definition of HTTP_PROXY_ENV_VARS was here
2022-10-29_07:00:09.14032 {"timestamp":"2022-10-29T07:00:09.140Z","pid":61021,"message":"! Unable to load application: NameError: uninitialized constant WebHookService::GITLAB_EVENT_HEADER"}
2022-10-29_07:00:09.14067 bundler: failed to load command: puma (/opt/gitlab/embedded/bin/puma)
2022-10-29_07:00:09.15343 NameError: uninitialized constant WebHookService::GITLAB_EVENT_HEADER
2022-10-29_07:00:09.15347   /opt/gitlab/embedded/service/gitlab-rails/lib/api/triggers.rb:7:in `<class:Triggers>'
2022-10-29_07:00:09.15348   /opt/gitlab/embedded/service/gitlab-rails/lib/api/triggers.rb:4:in `<module:API>'
2022-10-29_07:00:09.15349   /opt/gitlab/embedded/service/gitlab-rails/lib/api/triggers.rb:3:in `<top (required)>'
2022-10-29_07:00:09.15350   /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/zeitwerk-2.5.3/lib/zeitwerk/kernel.rb:27:in `require'
2022-10-29_07:00:09.15383   /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/zeitwerk-2.5.3/lib/zeitwerk/kernel.rb:27:in `require'
2022-10-29_07:00:09.15385   /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/zeitwerk-2.5.3/lib/zeitwerk/loader/helpers.rb:95:in `const_get'
2022-10-29_07:00:09.15386   /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/zeitwerk-2.5.3/lib/zeitwerk/loader/helpers.rb:95:in `cget'
2022-10-29_07:00:09.15388   /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/zeitwerk-2.5.3/lib/zeitwerk/loader.rb:231:in `block (2 levels) in eager_load'
2022-10-29_07:00:09.15392   /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/zeitwerk-2.5.3/lib/zeitwerk/loader/helpers.rb:26:in `block in ls'
2022-10-29_07:00:09.15394   /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/zeitwerk-2.5.3/lib/zeitwerk/loader/helpers.rb:18:in `each_child'
2022-10-29_07:00:09.15396   /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/zeitwerk-2.5.3/lib/zeitwerk/loader/helpers.rb:18:in `ls'
2022-10-29_07:00:09.15397   /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/zeitwerk-2.5.3/lib/zeitwerk/loader.rb:226:in `block in eager_load'
2022-10-29_07:00:09.15398   /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/zeitwerk-2.5.3/lib/zeitwerk/loader.rb:211:in `synchronize'
2022-10-29_07:00:09.15410   /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/zeitwerk-2.5.3/lib/zeitwerk/loader.rb:211:in `eager_load'
2022-10-29_07:00:09.15411   /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/zeitwerk-2.5.3/lib/zeitwerk/loader.rb:311:in `each'
2022-10-29_07:00:09.15412   /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/zeitwerk-2.5.3/lib/zeitwerk/loader.rb:311:in `eager_load_all'
2022-10-29_07:00:09.15414   /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/railties-6.1.4.4/lib/rails/application/finisher.rb:133:in `block in <module:Finisher>'
2022-10-29_07:00:09.15415   /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/railties-6.1.4.4/lib/rails/initializable.rb:32:in `instance_exec'
2022-10-29_07:00:09.15416   /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/railties-6.1.4.4/lib/rails/initializable.rb:32:in `run'
2022-10-29_07:00:09.15417   /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/railties-6.1.4.4/lib/rails/initializable.rb:61:in `block in run_initializers'
2022-10-29_07:00:09.15418   /opt/gitlab/embedded/lib/ruby/2.7.0/tsort.rb:228:in `block in tsort_each'
2022-10-29_07:00:09.15419   /opt/gitlab/embedded/lib/ruby/2.7.0/tsort.rb:350:in `block (2 levels) in each_strongly_connected_component'
2022-10-29_07:00:09.15420   /opt/gitlab/embedded/lib/ruby/2.7.0/tsort.rb:431:in `each_strongly_connected_component_from'
2022-10-29_07:00:09.15421   /opt/gitlab/embedded/lib/ruby/2.7.0/tsort.rb:349:in `block in each_strongly_connected_component'
2022-10-29_07:00:09.15422   /opt/gitlab/embedded/lib/ruby/2.7.0/tsort.rb:347:in `each'
2022-10-29_07:00:09.15423   /opt/gitlab/embedded/lib/ruby/2.7.0/tsort.rb:347:in `call'
2022-10-29_07:00:09.15424   /opt/gitlab/embedded/lib/ruby/2.7.0/tsort.rb:347:in `each_strongly_connected_component'
2022-10-29_07:00:09.15425   /opt/gitlab/embedded/lib/ruby/2.7.0/tsort.rb:226:in `tsort_each'
2022-10-29_07:00:09.15426   /opt/gitlab/embedded/lib/ruby/2.7.0/tsort.rb:205:in `tsort_each'
2022-10-29_07:00:09.15427   /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/railties-6.1.4.4/lib/rails/initializable.rb:60:in `run_initializers'
2022-10-29_07:00:09.15428   /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/railties-6.1.4.4/lib/rails/application.rb:391:in `initialize!'
2022-10-29_07:00:09.15430   /opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:7:in `<top (required)>'
2022-10-29_07:00:09.15431   /opt/gitlab/embedded/service/gitlab-rails/config.ru:5:in `require'
2022-10-29_07:00:09.15432   /opt/gitlab/embedded/service/gitlab-rails/config.ru:5:in `block in <main>'
2022-10-29_07:00:09.15433   /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rack-2.2.3/lib/rack/builder.rb:116:in `eval'
2022-10-29_07:00:09.15434   /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rack-2.2.3/lib/rack/builder.rb:116:in `new_from_string'
2022-10-29_07:00:09.15435   /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rack-2.2.3/lib/rack/builder.rb:105:in `load_file'
2022-10-29_07:00:09.15436   /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rack-2.2.3/lib/rack/builder.rb:66:in `parse_file'
2022-10-29_07:00:09.15437   /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/puma-5.5.2/lib/puma/configuration.rb:345:in `load_rackup'
2022-10-29_07:00:09.15438   /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/puma-5.5.2/lib/puma/configuration.rb:267:in `app'
2022-10-29_07:00:09.15439   /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/puma-5.5.2/lib/puma/runner.rb:149:in `load_and_bind'
2022-10-29_07:00:09.15441   /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/puma-5.5.2/lib/puma/cluster.rb:339:in `run'
2022-10-29_07:00:09.15442   /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/puma-5.5.2/lib/puma/launcher.rb:181:in `run'
2022-10-29_07:00:09.15443   /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/puma-5.5.2/lib/puma/cli.rb:80:in `run'
2022-10-29_07:00:09.15444   /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/puma-5.5.2/bin/puma:10:in `<top (required)>'
2022-10-29_07:00:09.15445   /opt/gitlab/embedded/bin/puma:23:in `load'
2022-10-29_07:00:09.15446   /opt/gitlab/embedded/bin/puma:23:in `<top (required)>'
2022-10-29_07:00:09.96338 {"timestamp":"2022-10-29T07:00:09.963Z","pid":61057,"message":"Puma starting in cluster mode..."}

I then ran gitlab-ctl tail sidekiq:

{"severity":"INFO","time":"2022-10-29T07:02:12.277Z","message":"Starting cluster with 1 processes"}
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab.rb:47: warning: already initialized constant Gitlab::APP_DIRS_PATTERN
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab.rb:47: warning: previous definition of APP_DIRS_PATTERN was here
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab.rb:48: warning: already initialized constant Gitlab::VERSION
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab.rb:48: warning: previous definition of VERSION was here
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab.rb:49: warning: already initialized constant Gitlab::INSTALLATION_TYPE
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab.rb:49: warning: previous definition of INSTALLATION_TYPE was here
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab.rb:50: warning: already initialized constant Gitlab::HTTP_PROXY_ENV_VARS
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab.rb:50: warning: previous definition of HTTP_PROXY_ENV_VARS was here
2022-10-29T07:02:24.244Z pid=62145 tid=yk1 INFO: Listening on queues 
***LONG_LIST_OF_STRINGS***
{"severity":"INFO","time":"2022-10-29T07:02:24.246Z","message":"Booting Sidekiq 6.3.1 with redis options {:instrumentation_class=\u003eGitlab::Instrumentation::Redis::Queues, :path=\u003e\"/var/opt/gitlab/redis/redis.socket\", :namespace=\u003e\"resque:gitlab\"}"}
{"severity":"INFO","time":"2022-10-29T07:02:26.775Z","message":"GitLab reliable fetch activated!"}
{"severity":"INFO","time":"2022-10-29T07:02:26.887Z","message":"Cron Jobs - deleted job with name: update_all_mirrors_worker"}
{"severity":"INFO","time":"2022-10-29T07:02:26.889Z","message":"Cron Jobs - deleted job with name: geo_sidekiq_cron_config_worker"}
{"severity":"INFO","time":"2022-10-29T07:02:26.891Z","message":"Cron Jobs - add job with name: stuck_ci_jobs_worker"}
{"severity":"INFO","time":"2022-10-29T07:02:26.893Z","message":"Cron Jobs - add job with name: pipeline_schedule_worker"}
{"severity":"INFO","time":"2022-10-29T07:02:26.895Z","message":"Cron Jobs - add job with name: expire_build_artifacts_worker"}
{"severity":"INFO","time":"2022-10-29T07:02:26.897Z","message":"Cron Jobs - add job with name: ci_pipelines_expire_artifacts_worker"}
{"severity":"INFO","time":"2022-10-29T07:02:26.898Z","message":"Cron Jobs - add job with name: ci_schedule_delete_objects_worker"}
{"severity":"INFO","time":"2022-10-29T07:02:26.900Z","message":"Cron Jobs - add job with name: environments_auto_stop_cron_worker"}
{"severity":"INFO","time":"2022-10-29T07:02:26.902Z","message":"Cron Jobs - add job with name: environments_auto_delete_cron_worker"}
{"severity":"INFO","time":"2022-10-29T07:02:26.903Z","message":"Cron Jobs - add job with name: repository_check_worker"}
{"severity":"INFO","time":"2022-10-29T07:02:26.905Z","message":"Cron Jobs - add job with name: admin_email_worker"}
{"severity":"INFO","time":"2022-10-29T07:02:26.907Z","message":"Cron Jobs - add job with name: personal_access_tokens_expiring_worker"}
{"severity":"INFO","time":"2022-10-29T07:02:26.911Z","message":"Cron Jobs - add job with name: personal_access_tokens_expired_notification_worker"}
{"severity":"INFO","time":"2022-10-29T07:02:26.915Z","message":"Cron Jobs - add job with name: repository_archive_cache_worker"}
{"severity":"INFO","time":"2022-10-29T07:02:26.917Z","message":"Cron Jobs - add job with name: import_export_project_cleanup_worker"}
{"severity":"INFO","time":"2022-10-29T07:02:26.919Z","message":"Cron Jobs - add job with name: ci_archive_traces_cron_worker"}
{"severity":"INFO","time":"2022-10-29T07:02:26.921Z","message":"Cron Jobs - add job with name: requests_profiles_worker"}
{"severity":"INFO","time":"2022-10-29T07:02:26.923Z","message":"Cron Jobs - add job with name: remove_expired_members_worker"}
{"severity":"INFO","time":"2022-10-29T07:02:26.925Z","message":"Cron Jobs - add job with name: remove_expired_group_links_worker"}
{"severity":"INFO","time":"2022-10-29T07:02:26.927Z","message":"Cron Jobs - add job with name: remove_unaccepted_member_invites_worker"}
{"severity":"INFO","time":"2022-10-29T07:02:26.929Z","message":"Cron Jobs - add job with name: prune_old_events_worker"}
{"severity":"INFO","time":"2022-10-29T07:02:26.930Z","message":"Cron Jobs - add job with name: trending_projects_worker"}
{"severity":"INFO","time":"2022-10-29T07:02:26.932Z","message":"Cron Jobs - add job with name: remove_unreferenced_lfs_objects_worker"}
{"severity":"INFO","time":"2022-10-29T07:02:26.933Z","message":"Cron Jobs - add job with name: import_stuck_project_import_jobs"}
{"severity":"INFO","time":"2022-10-29T07:02:26.935Z","message":"Cron Jobs - add job with name: jira_import_stuck_jira_import_jobs"}
{"severity":"INFO","time":"2022-10-29T07:02:26.937Z","message":"Cron Jobs - add job with name: stuck_export_jobs_worker"}
{"severity":"INFO","time":"2022-10-29T07:02:26.938Z","message":"Cron Jobs - add job with name: gitlab_service_ping_worker"}
{"severity":"INFO","time":"2022-10-29T07:02:26.940Z","message":"Cron Jobs - add job with name: stuck_merge_jobs_worker"}
{"severity":"INFO","time":"2022-10-29T07:02:26.941Z","message":"Cron Jobs - add job with name: pages_domain_verification_cron_worker"}
{"severity":"INFO","time":"2022-10-29T07:02:26.943Z","message":"Cron Jobs - add job with name: pages_domain_removal_cron_worker"}
{"severity":"INFO","time":"2022-10-29T07:02:26.945Z","message":"Cron Jobs - add job with name: pages_domain_ssl_renewal_cron_worker"}
{"severity":"INFO","time":"2022-10-29T07:02:26.946Z","message":"Cron Jobs - add job with name: issue_due_scheduler_worker"}
{"severity":"INFO","time":"2022-10-29T07:02:26.968Z","message":"Cron Jobs - add job with name: metrics_dashboard_schedule_annotations_prune_worker"}
{"severity":"INFO","time":"2022-10-29T07:02:26.970Z","message":"Cron Jobs - add job with name: schedule_migrate_external_diffs_worker"}
{"severity":"INFO","time":"2022-10-29T07:02:26.972Z","message":"Cron Jobs - add job with name: namespaces_prune_aggregation_schedules_worker"}
{"severity":"INFO","time":"2022-10-29T07:02:26.974Z","message":"Cron Jobs - add job with name: container_expiration_policy_worker"}
{"severity":"INFO","time":"2022-10-29T07:02:26.975Z","message":"Cron Jobs - add job with name: image_ttl_group_policy_worker"}
{"severity":"INFO","time":"2022-10-29T07:02:26.977Z","message":"Cron Jobs - add job with name: cleanup_dependency_proxy_worker"}
{"severity":"INFO","time":"2022-10-29T07:02:26.979Z","message":"Cron Jobs - add job with name: cleanup_package_registry_worker"}
{"severity":"INFO","time":"2022-10-29T07:02:26.980Z","message":"Cron Jobs - add job with name: x509_issuer_crl_check_worker"}
{"severity":"INFO","time":"2022-10-29T07:02:26.982Z","message":"Cron Jobs - add job with name: users_create_statistics_worker"}
{"severity":"INFO","time":"2022-10-29T07:02:26.983Z","message":"Cron Jobs - add job with name: authorized_project_update_periodic_recalculate_worker"}
{"severity":"INFO","time":"2022-10-29T07:02:26.985Z","message":"Cron Jobs - add job with name: update_container_registry_info_worker"}
{"severity":"INFO","time":"2022-10-29T07:02:26.986Z","message":"Cron Jobs - add job with name: postgres_dynamic_partitions_manager"}
{"severity":"INFO","time":"2022-10-29T07:02:26.988Z","message":"Cron Jobs - add job with name: postgres_dynamic_partitions_dropper"}
{"severity":"INFO","time":"2022-10-29T07:02:26.990Z","message":"Cron Jobs - add job with name: ci_platform_metrics_update_cron_worker"}
{"severity":"INFO","time":"2022-10-29T07:02:26.991Z","message":"Cron Jobs - add job with name: analytics_usage_trends_count_job_trigger_worker"}
{"severity":"INFO","time":"2022-10-29T07:02:26.993Z","message":"Cron Jobs - add job with name: member_invitation_reminder_emails_worker"}
{"severity":"INFO","time":"2022-10-29T07:02:26.994Z","message":"Cron Jobs - add job with name: schedule_merge_request_cleanup_refs_worker"}
{"severity":"INFO","time":"2022-10-29T07:02:26.996Z","message":"Cron Jobs - add job with name: manage_evidence_worker"}
{"severity":"INFO","time":"2022-10-29T07:02:26.998Z","message":"Cron Jobs - add job with name: user_status_cleanup_batch_worker"}
{"severity":"INFO","time":"2022-10-29T07:02:26.999Z","message":"Cron Jobs - add job with name: ssh_keys_expired_notification_worker"}
{"severity":"INFO","time":"2022-10-29T07:02:27.001Z","message":"Cron Jobs - add job with name: namespaces_in_product_marketing_emails_worker"}
{"severity":"INFO","time":"2022-10-29T07:02:27.002Z","message":"Cron Jobs - add job with name: ssh_keys_expiring_soon_notification_worker"}
{"severity":"INFO","time":"2022-10-29T07:02:27.004Z","message":"Cron Jobs - add job with name: users_deactivate_dormant_users_worker"}
{"severity":"INFO","time":"2022-10-29T07:02:27.005Z","message":"Cron Jobs - add job with name: ci_delete_unit_tests_worker"}
{"severity":"INFO","time":"2022-10-29T07:02:27.007Z","message":"Cron Jobs - add job with name: batched_background_migrations_worker"}
{"severity":"INFO","time":"2022-10-29T07:02:27.008Z","message":"Cron Jobs - add job with name: issues_reschedule_stuck_issue_rebalances"}
{"severity":"INFO","time":"2022-10-29T07:02:27.010Z","message":"Cron Jobs - add job with name: clusters_integrations_check_prometheus_health_worker"}
{"severity":"INFO","time":"2022-10-29T07:02:27.011Z","message":"Cron Jobs - add job with name: analytics_devops_adoption_create_all_snapshots_worker"}
{"severity":"INFO","time":"2022-10-29T07:02:27.013Z","message":"Cron Jobs - add job with name: active_user_count_threshold_worker"}
{"severity":"INFO","time":"2022-10-29T07:02:27.014Z","message":"Cron Jobs - add job with name: adjourned_group_deletion_worker"}
{"severity":"INFO","time":"2022-10-29T07:02:27.016Z","message":"Cron Jobs - add job with name: clear_shared_runners_minutes_worker"}
{"severity":"INFO","time":"2022-10-29T07:02:27.018Z","message":"Cron Jobs - add job with name: adjourned_projects_deletion_cron_worker"}
{"severity":"INFO","time":"2022-10-29T07:02:27.019Z","message":"Cron Jobs - add job with name: geo_verification_cron_worker"}
{"severity":"INFO","time":"2022-10-29T07:02:27.020Z","message":"Cron Jobs - add job with name: geo_sync_timeout_cron_worker"}
{"severity":"INFO","time":"2022-10-29T07:02:27.022Z","message":"Cron Jobs - add job with name: geo_secondary_usage_data_cron_worker"}
{"severity":"INFO","time":"2022-10-29T07:02:27.023Z","message":"Cron Jobs - add job with name: geo_file_download_dispatch_worker"}
{"severity":"INFO","time":"2022-10-29T07:02:27.025Z","message":"Cron Jobs - add job with name: geo_registry_sync_worker"}
{"severity":"INFO","time":"2022-10-29T07:02:27.026Z","message":"Cron Jobs - add job with name: geo_metrics_update_worker"}
{"severity":"INFO","time":"2022-10-29T07:02:27.028Z","message":"Cron Jobs - add job with name: geo_prune_event_log_worker"}
{"severity":"INFO","time":"2022-10-29T07:02:27.029Z","message":"Cron Jobs - add job with name: geo_repository_sync_worker"}
{"severity":"INFO","time":"2022-10-29T07:02:27.031Z","message":"Cron Jobs - add job with name: geo_secondary_registry_consistency_worker"}
{"severity":"INFO","time":"2022-10-29T07:02:27.033Z","message":"Cron Jobs - add job with name: geo_repository_verification_primary_batch_worker"}
{"severity":"INFO","time":"2022-10-29T07:02:27.034Z","message":"Cron Jobs - add job with name: geo_repository_verification_secondary_scheduler_worker"}
{"severity":"INFO","time":"2022-10-29T07:02:27.036Z","message":"Cron Jobs - add job with name: geo_container_repository_sync_worker"}
{"severity":"INFO","time":"2022-10-29T07:02:27.037Z","message":"Cron Jobs - add job with name: historical_data_worker"}
{"severity":"INFO","time":"2022-10-29T07:02:27.039Z","message":"Cron Jobs - add job with name: incident_sla_exceeded_check_worker"}
{"severity":"INFO","time":"2022-10-29T07:02:27.040Z","message":"Cron Jobs - add job with name: incident_management_persist_oncall_rotation_worker"}
{"severity":"INFO","time":"2022-10-29T07:02:27.042Z","message":"Cron Jobs - add job with name: incident_management_schedule_escalation_check_worker"}
{"severity":"INFO","time":"2022-10-29T07:02:27.044Z","message":"Cron Jobs - add job with name: import_software_licenses_worker"}
{"severity":"INFO","time":"2022-10-29T07:02:27.045Z","message":"Cron Jobs - add job with name: ldap_group_sync_worker"}
{"severity":"INFO","time":"2022-10-29T07:02:27.047Z","message":"Cron Jobs - add job with name: ldap_sync_worker"}
{"severity":"INFO","time":"2022-10-29T07:02:27.049Z","message":"Cron Jobs - add job with name: pseudonymizer_worker"}
{"severity":"INFO","time":"2022-10-29T07:02:27.050Z","message":"Cron Jobs - add job with name: update_max_seats_used_for_gitlab_com_subscriptions_worker"}
{"severity":"INFO","time":"2022-10-29T07:02:27.052Z","message":"Cron Jobs - add job with name: elastic_index_bulk_cron_worker"}
{"severity":"INFO","time":"2022-10-29T07:02:27.054Z","message":"Cron Jobs - add job with name: elastic_index_initial_bulk_cron_worker"}
{"severity":"INFO","time":"2022-10-29T07:02:27.055Z","message":"Cron Jobs - add job with name: elastic_cluster_reindexing_cron_worker"}
{"severity":"INFO","time":"2022-10-29T07:02:27.057Z","message":"Cron Jobs - add job with name: elastic_remove_expired_namespace_subscriptions_from_index_cron_worker"}
{"severity":"INFO","time":"2022-10-29T07:02:27.059Z","message":"Cron Jobs - add job with name: elastic_migration_worker"}
{"severity":"INFO","time":"2022-10-29T07:02:27.061Z","message":"Cron Jobs - add job with name: sync_seat_link_worker"}
{"severity":"INFO","time":"2022-10-29T07:02:27.063Z","message":"Cron Jobs - add job with name: network_policy_metrics_worker"}
{"severity":"INFO","time":"2022-10-29T07:02:27.064Z","message":"Cron Jobs - add job with name: iterations_update_status_worker"}
{"severity":"INFO","time":"2022-10-29T07:02:27.066Z","message":"Cron Jobs - add job with name: iterations_generator_worker"}
{"severity":"INFO","time":"2022-10-29T07:02:27.068Z","message":"Cron Jobs - add job with name: vulnerability_statistics_schedule_worker"}
{"severity":"INFO","time":"2022-10-29T07:02:27.069Z","message":"Cron Jobs - add job with name: vulnerability_historical_statistics_deletion_worker"}
{"severity":"INFO","time":"2022-10-29T07:02:27.071Z","message":"Cron Jobs - add job with name: security_create_orchestration_policy_worker"}
{"severity":"INFO","time":"2022-10-29T07:02:27.073Z","message":"Cron Jobs - add job with name: security_orchestration_policy_rule_schedule_worker"}
{"severity":"INFO","time":"2022-10-29T07:02:27.074Z","message":"Cron Jobs - add job with name: app_sec_dast_profile_schedule_worker"}
{"severity":"INFO","time":"2022-10-29T07:02:27.076Z","message":"Cron Jobs - add job with name: loose_foreign_keys_cleanup_worker"}
{"severity":"INFO","time":"2022-10-29T07:02:27.088Z","message":"Cron Jobs - add job with name: update_all_mirrors_worker"}
{"severity":"INFO","time":"2022-10-29T07:02:27.113Z","message":"Cron Jobs - add job with name: geo_sidekiq_cron_config_worker"}
{"severity":"WARN","time":"2022-10-29T07:02:28.443Z","message":"Creating scope :without_statuses. Overwriting existing method CommitStatus.without_statuses."}
{"severity":"WARN","time":"2022-10-29T07:02:28.484Z","message":"Creating scope :without_statuses. Overwriting existing method Ci::Pipeline.without_statuses."}
{"severity":"WARN","time":"2022-10-29T07:02:28.669Z","message":"Creating scope :in_pipelines. Overwriting existing method Ci::Build.in_pipelines."}
{"severity":"WARN","time":"2022-10-29T07:02:29.267Z","message":"Creating scope :of_projects. Overwriting existing method MergeRequest.of_projects."}
{"severity":"WARN","time":"2022-10-29T07:02:29.268Z","message":"Creating scope :join_project. Overwriting existing method MergeRequest.join_project."}
{"severity":"WARN","time":"2022-10-29T07:02:29.268Z","message":"Creating scope :references_project. Overwriting existing method MergeRequest.references_project."}
{"severity":"WARN","time":"2022-10-29T07:02:29.297Z","message":"Creating scope :system. Overwriting existing method Note.system."}
{"severity":"WARN","time":"2022-10-29T07:02:29.569Z","message":"Creating scope :closed. Overwriting existing method Iteration.closed."}
{"severity":"WARN","time":"2022-10-29T07:02:29.569Z","message":"Creating scope :within_timeframe. Overwriting existing method Iteration.within_timeframe."}
{"severity":"WARN","time":"2022-10-29T07:02:29.899Z","message":"Creating scope :opened. Overwriting existing method Epic.opened."}
{"severity":"WARN","time":"2022-10-29T07:02:29.900Z","message":"Creating scope :closed. Overwriting existing method Epic.closed."}
{"severity":"WARN","time":"2022-10-29T07:02:30.053Z","message":"Creating scope :order_created_desc. Overwriting existing method Packages::Package.order_created_desc."}
{"severity":"WARN","time":"2022-10-29T07:02:30.054Z","message":"Creating scope :order_name_desc. Overwriting existing method Packages::Package.order_name_desc."}
{"severity":"WARN","time":"2022-10-29T07:02:31.595Z","message":"Creating scope :open. Overwriting existing method AlertManagement::Alert.open."}
{"severity":"WARN","time":"2022-10-29T07:02:31.621Z","message":"Creating scope :open. Overwriting existing method IncidentManagement::IssuableEscalationStatus.open."}
{"severity":"WARN","time":"2022-10-29T07:02:31.677Z","message":"Creating scope :with_associations. Overwriting existing method Board.with_associations."}
{"severity":"WARN","time":"2022-10-29T07:02:31.729Z","message":"Creating scope :movable. Overwriting existing method Boards::EpicList.movable."}
{"severity":"WARN","time":"2022-10-29T07:02:32.078Z","message":"Creating scope :order_name_asc. Overwriting existing method Label.order_name_asc."}
{"severity":"WARN","time":"2022-10-29T07:02:32.079Z","message":"Creating scope :order_name_desc. Overwriting existing method Label.order_name_desc."}
{"severity":"WARN","time":"2022-10-29T07:02:34.602Z","message":"Creating scope :with_logging_info. Overwriting existing method PagesDomain.with_logging_info."}
{"severity":"WARN","time":"2022-10-29T07:02:34.841Z","message":"Creating scope :system. Overwriting existing method PrometheusMetric.system."}
{"severity":"WARN","time":"2022-10-29T07:02:34.856Z","message":"Creating scope :order_name_asc. Overwriting existing method Member.order_name_asc."}
{"severity":"WARN","time":"2022-10-29T07:02:34.857Z","message":"Creating scope :order_name_desc. Overwriting existing method Member.order_name_desc."}
{"severity":"WARN","time":"2022-10-29T07:02:34.878Z","message":"Creating scope :open. Overwriting existing method ExternalPullRequest.open."}
{"severity":"WARN","time":"2022-10-29T07:02:34.975Z","message":"Creating scope :with_user. Overwriting existing method GroupMember.with_user."}
{"severity":"WARN","time":"2022-10-29T07:02:34.977Z","message":"Creating scope :reporters. Overwriting existing method GroupMember.reporters."}
{"severity":"WARN","time":"2022-10-29T07:02:34.977Z","message":"Creating scope :guests. Overwriting existing method GroupMember.guests."}
{"severity":"WARN","time":"2022-10-29T07:02:35.136Z","message":"Creating scope :extended. Overwriting existing method GitlabSubscription.extended."}
{"severity":"WARN","time":"2022-10-29T07:02:35.519Z","message":"Creating scope :database. Overwriting existing method Ci::BuildTraceChunk.database."}
{"severity":"WARN","time":"2022-10-29T07:02:35.594Z","message":"Creating scope :without_statuses. Overwriting existing method Ci::Stage.without_statuses."}
{"severity":"WARN","time":"2022-10-29T07:02:35.595Z","message":"Creating scope :created. Overwriting existing method Ci::Stage.created."}
{"severity":"WARN","time":"2022-10-29T07:02:35.604Z","message":"Creating scope :pending. Overwriting existing method Ci::Stage.pending."}
{"severity":"WARN","time":"2022-10-29T07:02:35.604Z","message":"Creating scope :running. Overwriting existing method Ci::Stage.running."}
{"severity":"WARN","time":"2022-10-29T07:02:35.605Z","message":"Creating scope :success. Overwriting existing method Ci::Stage.success."}
{"severity":"WARN","time":"2022-10-29T07:02:35.605Z","message":"Creating scope :failed. Overwriting existing method Ci::Stage.failed."}
{"severity":"WARN","time":"2022-10-29T07:02:35.605Z","message":"Creating scope :canceled. Overwriting existing method Ci::Stage.canceled."}
{"severity":"WARN","time":"2022-10-29T07:02:35.606Z","message":"Creating scope :skipped. Overwriting existing method Ci::Stage.skipped."}
{"severity":"WARN","time":"2022-10-29T07:02:35.606Z","message":"Creating scope :manual. Overwriting existing method Ci::Stage.manual."}
{"severity":"WARN","time":"2022-10-29T07:02:35.606Z","message":"Creating scope :scheduled. Overwriting existing method Ci::Stage.scheduled."}
{"severity":"WARN","time":"2022-10-29T07:02:35.607Z","message":"Creating scope :preparing. Overwriting existing method Ci::Stage.preparing."}
{"severity":"WARN","time":"2022-10-29T07:02:35.607Z","message":"Creating scope :waiting_for_resource. Overwriting existing method Ci::Stage.waiting_for_resource."}
{"severity":"INFO","time":"2022-10-29T07:02:37.310Z","message":"A worker terminated, shutting down the cluster"}
{"severity":"INFO","time":"2022-10-29T07:02:38.223Z","message":"Starting cluster with 1 processes"}

gitlab-rake gitlab:check SANITIZE=true shows:

/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab.rb:47: warning: already initialized constant Gitlab::APP_DIRS_PATTERN
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab.rb:47: warning: previous definition of APP_DIRS_PATTERN was here
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab.rb:48: warning: already initialized constant Gitlab::VERSION
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab.rb:48: warning: previous definition of VERSION was here
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab.rb:49: warning: already initialized constant Gitlab::INSTALLATION_TYPE
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab.rb:49: warning: previous definition of INSTALLATION_TYPE was here
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab.rb:50: warning: already initialized constant Gitlab::HTTP_PROXY_ENV_VARS
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab.rb:50: warning: previous definition of HTTP_PROXY_ENV_VARS was here
Checking GitLab subtasks ...

Checking GitLab Shell ...

GitLab Shell: ... GitLab Shell version >= 13.22.2 ? ... OK (13.22.2)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Internal API available: FAILED - Internal API error (502)
gitlab-shell self-check failed
  Try fixing it:
  Make sure GitLab is running;
  Check the gitlab-shell configuration file:
  sudo -u git -H editor /opt/gitlab/embedded/service/gitlab-shell/config.yml
  Please fix the error above and rerun the checks.

Checking GitLab Shell ... Finished

Checking Gitaly ...

Gitaly: ... default ... OK

Checking Gitaly ... Finished

Checking Sidekiq ...

Sidekiq: ... Running? ... no
  Try fixing it:
  sudo -u git -H RAILS_ENV=production bin/background_jobs start
  For more information see:
  doc/install/installation.md in section "Install Init Script"
  see log/sidekiq.log for possible errors
  Please fix the error above and rerun the checks.

Checking Sidekiq ... Finished

Checking Incoming Email ...

Incoming Email: ... Reply by email is disabled in config/gitlab.yml

Checking Incoming Email ... Finished

Checking LDAP ...

LDAP: ... Server: ldapmain
not verifying SSL hostname of LDAPS server 'X'
LDAP authentication... Success
LDAP users with access to your GitLab server (only showing the first 100 results)
	User output sanitized. Found 57 users of 100 limit.

Checking LDAP ... Finished

Checking GitLab App ...

Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Systemd unit files or init script exist? ... skipped (omnibus-gitlab has neither init script nor systemd units)
Systemd unit files or init script up-to-date? ... skipped (omnibus-gitlab has neither init script nor systemd units)
Projects have namespace: ...
2/1 ... yes
13/2 ... yes
9/3 ... yes
9/4 ... yes
9/6 ... yes
9/7 ... yes
9/8 ... yes
13/9 ... yes
6/10 ... yes
6/11 ... yes
10/12 ... yes
9/13 ... yes
9/14 ... yes
9/15 ... yes
9/16 ... yes
9/17 ... yes
9/18 ... yes
9/19 ... yes
9/20 ... yes
9/21 ... yes
9/23 ... yes
9/24 ... yes
9/25 ... yes
9/26 ... yes
9/27 ... yes
9/28 ... yes
9/29 ... yes
9/30 ... yes
9/31 ... yes
9/32 ... yes
9/33 ... yes
9/34 ... yes
9/35 ... yes
9/36 ... yes
9/37 ... yes
9/38 ... yes
9/39 ... yes
9/40 ... yes
9/41 ... yes
9/42 ... yes
9/43 ... yes
9/44 ... yes
9/45 ... yes
9/46 ... yes
9/47 ... yes
9/48 ... yes
9/49 ... yes
9/50 ... yes
9/51 ... yes
9/52 ... yes
9/53 ... yes
9/54 ... yes
9/55 ... yes
9/56 ... yes
9/57 ... yes
9/58 ... yes
9/59 ... yes
9/60 ... yes
9/61 ... yes
9/62 ... yes
9/63 ... yes
9/64 ... yes
9/65 ... yes
9/66 ... yes
9/67 ... yes
9/68 ... yes
9/69 ... yes
9/70 ... yes
9/71 ... yes
9/72 ... yes
9/73 ... yes
9/76 ... yes
9/77 ... yes
9/78 ... yes
9/79 ... yes
9/80 ... yes
9/81 ... yes
9/82 ... yes
9/83 ... yes
9/84 ... yes
9/85 ... yes
9/86 ... yes
9/87 ... yes
9/88 ... yes
9/89 ... yes
9/91 ... yes
9/92 ... yes
9/93 ... yes
9/94 ... yes
9/95 ... yes
9/96 ... yes
9/97 ... yes
9/98 ... yes
9/99 ... yes
9/100 ... yes
9/101 ... yes
9/102 ... yes
9/103 ... yes
9/104 ... yes
9/105 ... yes
9/106 ... yes
9/107 ... yes
9/108 ... yes
9/109 ... yes
9/110 ... yes
9/111 ... yes
9/112 ... yes
9/113 ... yes
9/114 ... yes
9/115 ... yes
9/116 ... yes
7/117 ... yes
14/119 ... yes
9/120 ... yes
9/121 ... yes
9/122 ... yes
13/123 ... yes
7/124 ... yes
7/125 ... yes
7/126 ... yes
7/127 ... yes
7/128 ... yes
7/129 ... yes
7/130 ... yes
7/131 ... yes
7/132 ... yes
12/133 ... yes
12/134 ... yes
13/135 ... yes
13/136 ... yes
9/137 ... yes
13/138 ... yes
13/139 ... yes
13/140 ... yes
13/141 ... yes
10/142 ... yes
6/143 ... yes
12/144 ... yes
9/145 ... yes
7/146 ... yes
7/147 ... yes
14/148 ... yes
6/149 ... yes
9/150 ... yes
9/151 ... yes
16/152 ... yes
16/153 ... yes
14/154 ... yes
7/155 ... yes
10/156 ... yes
12/158 ... yes
9/159 ... yes
9/160 ... yes
7/161 ... yes
19/162 ... yes
19/163 ... yes
19/164 ... yes
20/165 ... yes
9/166 ... yes
9/167 ... yes
20/168 ... yes
9/169 ... yes
20/170 ... yes
20/171 ... yes
20/172 ... yes
19/173 ... yes
10/174 ... yes
7/175 ... yes
13/178 ... yes
13/179 ... yes
13/180 ... yes
14/181 ... yes
30/182 ... yes
30/183 ... yes
13/184 ... yes
30/185 ... yes
30/186 ... yes
12/187 ... yes
36/189 ... yes
7/190 ... yes
13/191 ... yes
37/192 ... yes
37/193 ... yes
8/194 ... yes
9/196 ... yes
9/197 ... yes
9/198 ... yes
9/199 ... yes
9/200 ... yes
9/201 ... yes
9/202 ... yes
9/204 ... yes
9/205 ... yes
13/206 ... yes
9/207 ... yes
37/208 ... yes
7/209 ... yes
13/210 ... yes
13/211 ... yes
13/212 ... yes
7/213 ... yes
13/214 ... yes
9/215 ... yes
9/216 ... yes
9/217 ... yes
9/218 ... yes
9/219 ... yes
19/220 ... yes
50/221 ... yes
13/222 ... yes
13/223 ... yes
7/225 ... yes
21/226 ... yes
30/227 ... yes
30/228 ... yes
30/229 ... yes
30/230 ... yes
50/231 ... yes
13/232 ... yes
36/234 ... yes
20/235 ... yes
9/236 ... yes
9/237 ... yes
9/238 ... yes
9/239 ... yes
9/240 ... yes
14/241 ... yes
14/242 ... yes
14/243 ... yes
20/244 ... yes
9/245 ... yes
20/246 ... yes
13/247 ... yes
9/248 ... yes
7/249 ... yes
13/250 ... yes
20/251 ... yes
7/252 ... yes
7/253 ... yes
7/254 ... yes
72/255 ... yes
72/256 ... yes
72/257 ... yes
72/258 ... yes
72/259 ... yes
72/260 ... yes
72/261 ... yes
72/262 ... yes
72/263 ... yes
7/264 ... yes
106/265 ... yes
30/266 ... yes
50/267 ... yes
7/268 ... yes
13/269 ... yes
13/270 ... yes
7/272 ... yes
7/273 ... yes
7/274 ... yes
7/276 ... yes
13/277 ... yes
9/278 ... yes
20/279 ... yes
57/280 ... yes
57/281 ... yes
20/282 ... yes
7/283 ... yes
20/284 ... yes
107/285 ... yes
13/286 ... yes
57/287 ... yes
20/288 ... yes
13/289 ... yes
9/290 ... yes
7/291 ... yes
7/292 ... yes
50/293 ... yes
115/294 ... yes
115/295 ... yes
115/296 ... yes
115/297 ... yes
115/298 ... yes
115/299 ... yes
115/300 ... yes
50/301 ... yes
9/302 ... yes
50/303 ... yes
9/304 ... yes
37/306 ... yes
37/307 ... yes
37/308 ... yes
37/309 ... yes
9/310 ... yes
7/311 ... yes
7/312 ... yes
9/313 ... yes
20/314 ... yes
50/315 ... yes
13/316 ... yes
9/318 ... yes
20/319 ... yes
14/323 ... yes
20/324 ... yes
7/325 ... yes
136/327 ... yes
136/328 ... yes
20/329 ... yes
136/330 ... yes
7/331 ... yes
9/332 ... yes
14/333 ... yes
7/334 ... yes
36/335 ... yes
57/336 ... yes
7/337 ... yes
13/338 ... yes
20/339 ... yes
20/340 ... yes
7/341 ... yes
9/342 ... yes
19/343 ... yes
19/344 ... yes
8/345 ... yes
7/346 ... yes
9/347 ... yes
137/348 ... yes
137/349 ... yes
137/350 ... yes
137/351 ... yes
137/352 ... yes
137/353 ... yes
137/354 ... yes
137/355 ... yes
137/356 ... yes
6/357 ... yes
7/358 ... yes
9/359 ... yes
13/360 ... yes
9/361 ... yes
7/362 ... yes
7/363 ... yes
7/364 ... yes
7/365 ... yes
26/366 ... yes
13/367 ... yes
9/368 ... yes
57/369 ... yes
7/370 ... yes
80/371 ... yes
78/372 ... yes
12/373 ... yes
7/374 ... yes
7/375 ... yes
7/376 ... yes
20/377 ... yes
13/378 ... yes
30/379 ... yes
9/380 ... yes
14/381 ... yes
150/382 ... yes
9/383 ... yes
30/384 ... yes
14/385 ... yes
107/386 ... yes
107/387 ... yes
20/388 ... yes
9/389 ... yes
107/390 ... yes
14/391 ... yes
57/392 ... yes
7/393 ... yes
7/394 ... yes
7/395 ... yes
13/396 ... yes
7/397 ... yes
7/398 ... yes
7/399 ... yes
57/400 ... yes
9/401 ... yes
7/402 ... yes
7/403 ... yes
7/404 ... yes
14/405 ... yes
147/406 ... yes
9/408 ... yes
9/409 ... yes
Redis version >= 5.0.0? ... yes
Ruby version >= 2.7.2 ? ... yes (2.7.5)
Git version >= 2.33.0 ? ... yes (2.33.1)
Git user has default SSH configuration? ... yes
Active users: ... 99
Is authorized keys file accessible? ... yes
GitLab configured to store new projects in hashed storage? ... yes
All projects are in hashed storage? ... yes
Elasticsearch version 7.x (6.4 - 6.x deprecated to be removed in 13.8)? ... skipped (elasticsearch is disabled)

Checking GitLab App ... Finished


Checking GitLab subtasks ... Finished

gitlab-ctl status
run: alertmanager: (pid 1569) 296s; run: log: (pid 1553) 296s
run: gitaly: (pid 1574) 296s; run: log: (pid 1551) 296s
run: gitlab-exporter: (pid 1590) 295s; run: log: (pid 1557) 296s
run: gitlab-workhorse: (pid 1576) 296s; run: log: (pid 1552) 296s
run: grafana: (pid 1549) 296s; run: log: (pid 1541) 296s
run: logrotate: (pid 1550) 296s; run: log: (pid 1543) 296s
run: nginx: (pid 1571) 296s; run: log: (pid 1547) 296s
run: node-exporter: (pid 1546) 296s; run: log: (pid 1538) 296s
run: postgres-exporter: (pid 1591) 295s; run: log: (pid 1565) 296s
run: postgresql: (pid 1548) 296s; run: log: (pid 1540) 296s
run: prometheus: (pid 1580) 296s; run: log: (pid 1555) 296s
run: puma: (pid 4180) 21s; run: log: (pid 1556) 296s
run: redis: (pid 1578) 296s; run: log: (pid 1563) 296s
run: redis-exporter: (pid 1575) 296s; run: log: (pid 1559) 296s
run: sidekiq: (pid 4204) 1s; run: log: (pid 1561) 296s

sudo gitlab-rake zeitwerk:check
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab.rb:47: warning: already initialized constant Gitlab::APP_DIRS_PATTERN
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab.rb:47: warning: previous definition of APP_DIRS_PATTERN was here
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab.rb:48: warning: already initialized constant Gitlab::VERSION
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab.rb:48: warning: previous definition of VERSION was here
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab.rb:49: warning: already initialized constant Gitlab::INSTALLATION_TYPE
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab.rb:49: warning: previous definition of INSTALLATION_TYPE was here
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab.rb:50: warning: already initialized constant Gitlab::HTTP_PROXY_ENV_VARS
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab.rb:50: warning: previous definition of HTTP_PROXY_ENV_VARS was here
Hold on, I am eager loading the application.
rake aborted!
NameError: uninitialized constant WebHookService::GITLAB_EVENT_HEADER
/opt/gitlab/embedded/service/gitlab-rails/lib/api/triggers.rb:7:in `<class:Triggers>'
/opt/gitlab/embedded/service/gitlab-rails/lib/api/triggers.rb:4:in `<module:API>'
/opt/gitlab/embedded/service/gitlab-rails/lib/api/triggers.rb:3:in `<top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => zeitwerk:check
(See full trace by running task with --trace)

What does /opt/gitlab/embedded/service/gitlab-shell/config.yml look like?

Does anything change if you run gitlab-ctl reconfigure?

Thanks a lot for your reply, highly appreciated. But I was so frustrated that I dropped GitLab and switched to Gitea in the meanwhile.