After upgrade from 15.11.5 to 16.0,GitLab CE down, cpu 100%

After upgrade from 15.11.5 to 16.0,GitLab CE down, cpu 100%

Check using top or htop to see what process is consuming all the CPU. Also, how many CPU does the VM/server have? How much ram? Minimum recommended is 4cpu and 8gb ram.

top cpu useage by git bundle. 4CPU 8GB

6CPU, 24GB Same thing after updating to 16.0

Снимок экрана 2023-05-23 1214531912×600 36.1 KB

Hi,
founding problem, in gitlab.rb, we have configured OAuth2_generic settings for WeWork/Wechat login, when disable the settings. It’s ok. I don’t know why, before 16.0, it’s worked. Is there any change for OAuth2_Generic in v16.0?

After update migrations will run, but the web interface should be accessible. DB Migrations will use CPU resources. Usually after restart, for the first 1 - 2 mins the web interface is inaccessible.

of course, i know. I update from 9,to 10,11,12,13,14,15, all are ok. But, when 16,failed

Hi,

same problem here, but deactivating omniauth did not fix the problem.

The server has 6 CPU and 12 GB RAM.

In the sidekiq/current log, it gives an error:
incompatible character encodings: US-ASCII and UTF-8 (Encoding::CompatibilityError)
…however I don’t know if that’s related.

We updated from 15.11 to 16.0 via the launchpad ppa on ubuntu.

Any help?

Hi,
Is there any cue to investigate? To check which log files?Any help?

Same issue here. I’ve been trying the latest 16.0.1 version but it still consumes all computer resources within few minutes.

Me, too. Hopefully a solution is available soon.

Hi,
Same issue here. 8Gb 4cpu debian 11.7 under hyperV
Just after upgrading loads of bundle process fill up memory. Number keep growing with time (waited about an hour before applying snapshot)
We have two omniauth saml sso configuration. I can easily make tests overnight if needed.

after update 16.0 to 16.0.1. No changes, cpu 100% 50 min

Снимок экрана 2023-05-24 0955041905×954 55.9 KB

Hello,

Same over here, when upgrading my omnibus Gitlab from 15.11.5 to 16.0.1, CPU and RAM 100% with too many sidekiq processes and Gitlab broke up.

On the other hand, I see some repetitive line on sidekiq current log like “/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/activesupport-6.1.7.2/lib/active_support/configuration_file.rb:40:in `include?': incompatible character encodings: US-ASCII and UTF-8 (Encoding::CompatibilityError)”

Any ideas?

Hi,
All, found the problem and solved,for 16.0, in the gitlab.rb, all of characters muast be ASCII character, cannot contain other language characters, the problem ruby files is:
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/activesupport-6.1.7.2/lib/active_support/configuration_file.rb:40:in include?': incompatible character encodings: US-ASCII and UTF-8 (Encoding::CompatibilityError) from /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/activesupport-6.1.7.2/lib/active_support/configuration_file.rb:40:in block in read’
from internal:kernel:90:in tap' from /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/activesupport-6.1.7.2/lib/active_support/configuration_file.rb:39:in read’
from /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/activesupport-6.1.7.2/lib/active_support/configuration_file.rb:14:in initialize' from /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/activesupport-6.1.7.2/lib/active_support/configuration_file.rb:18:in new’
from /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/activesupport-6.1.7.2/lib/active_support/configuration_file.rb:18:in parse' from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab_settings/settings.rb:17:in reload!’
from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab_settings/settings.rb:27:in method_missing' from /opt/gitlab/embedded/service/gitlab-rails/sidekiq_cluster/cli.rb:202:in sidekiq_exporter_enabled?’
from /opt/gitlab/embedded/service/gitlab-rails/sidekiq_cluster/cli.rb:210:in metrics_server_enabled?' from /opt/gitlab/embedded/service/gitlab-rails/sidekiq_cluster/cli.rb:188:in start_metrics_server’
from /opt/gitlab/embedded/service/gitlab-rails/sidekiq_cluster/cli.rb:162:in start_and_supervise_workers' from /opt/gitlab/embedded/service/gitlab-rails/sidekiq_cluster/cli.rb:136:in run’
from /opt/gitlab/embedded/service/gitlab-rails/bin/sidekiq-cluster:11:in `’

Hi,

Same here, upgrade from 15.11.5 to 16.0.0, CPU and RAM 100% with a lot af “bundle” process, caused by sidekiq. The VM has 4 CPUs and 8 GB of memory, Ubuntu 20.04.6 LTS up to date. We increase to 12 CPUs and 64 GB of memory, this allow us to use Gitlab but we still have all CPUs @ 100% and around 18 GB of memory used…
We did the upgrade to Gitlab 16.0.1 but it’s just a security release, it does not fix this issue.

[UPDATE]
I confirm the previous answer from @softboy99 : remove non ASCII characters in /etc/gitlab/gitlab.rb seems to solve the problem (after gitlab-ctl reconfigure of course).

@softboy99 Thanks! This worked!

Hint for all struggling:
to check where the culprit is, use
grep --color='auto' -P -n "[\x80-\xFF]" /etc/gitlab/gitlab.rb
(Source)

In our case it was a “§” character.

@softboy99 Your solution worked!! Thanks a lot!

Removing a German Umlaut (“ü”) from the configuration file was the solution for me, too.

1. Edit /etc/gitlab/gitlab.rb and remove/replace the Umlaut.
2. Call gitlab-ctl reconfigure.
3. Reboot sudo reboot my system, just to be 100% sure.

Now it is working again. Thanks to all of you here helping

I’m curious as to why there were these characters in the config file? In which place exactly? external_url? Somewhere else? What was the reason for including them or were they there by mistake eg: like copying/sending the file from Windows to Linux? If for the latter, using dos2unix can convert the file to be read correctly in Linux. Usually removes the special characters from the end of line related to line-feed, etc.

If you were expecting these characters to work, or the potential for Gitlab to support these characters, perhaps it would be ideal to open an issue here: Issues · GitLab.org / GitLab · GitLab (assuming an issue doesn’t already exist for it).