Fixed: Upgrade from 15.11 to 16.0.1 generates endless cpu load and full ram / SWAP

Hello everyone,

The last updates before went absolutely smoothly.

I have now carried out a test run of the update from Gitlab 15.11 to version 16.0.1 and here I have an endless CPU load that ebbs after a short time for a few seconds and then goes back to 100% for all cores.

After about 16 minutes there is a short break in the load, and then it goes up to 100% again and again.

Each time it is the process

/opt/gitlab/embedded/bin/sidekiq -c20 -eproduction -t25 -gqueues:authorised_project_update:authorised_project_update_project_recalculate,authorised_project_update:authorised_project_update_project_recalculate_per_user,authorised_project_update: authorized_project_update_user_refresh_from_replica,authorized_project_update:authorized_project_update_user_refresh_over_user_range,authorized_project_update:authorized_project_update_user_refresh_with_low_urgency,auto_devops:auto_devops_disable,auto_merge: auto_merge_process,batched_background_migrations:database_batched_background_migration_ci_execution,batched_background_migrations:database_batched_background_migration_main_execution,chaos:chaos_cpu_spin,chaos:chaos_db_spin,chaos:chaos_kill,chaos:chaos_leak_mem,chaos: chaos_sleep,cluster_agent:clusters_agents_delete_expired_events,cluster_agent:clusters_agents_notify_git_push,container_repository:cleanup_container_repository,container_repository:container_expiration_policies_cleanup_container_repository,container_rep

which occurs several times.
I know this behaviour in a simple form from previous versions.

Virtual machine:

By default, our Gitlab ran in a virtualised environment on Debian 11 Bullseye with 4 cores and 12 GB RAM, as I only use it for my configurations and ansible. It ran comfortably here for the last 2 years and felt perfectly at home.
I used packages.gitlab.com as Repository for Debian.

Only changes in gitlab.rb are:

  • External URL
  • SMTP-Server configuration
  • nginx enabled http2, gzip and added ssl-certificates

For the update, I expanded the system to 12 cores and 24GB RAM.
Here, the entire RAM and all cores are used again and again.
The RAM is then emptied briefly, only to fill up continuously afterwards and then fill the SWAP.

Tried so far:

  • Reboot
  • gitlab-ctl reconfigure with / or without reboot

Is there any idea on the subject here, or has anyone noticed this behaviour?

Thank you in advance and greetings

Christian

And of course something for the eyes, a screenshot :wink:

Some people have had similar problems, due to UTF characters in the gitlab.rb config file. You may wish to check some of the suggestions in this post:

other than the workaround linked, higher up in the post mentions removing the characters and replacing them with standard ascii characters. Of course, assuming that your problem is similar to this.

3 Likes

Thank you @iwalker

Sorry for overlooking the post.
This fixes this behaviour
Greetings and stay healthy

Christian

1 Like