Gitlab CE to EE upgrade

Hello everyone!

I did an upgrade from Gitlab CE 15.4.1-ce.0 to Gitlab EE 15.4.1-ce.0 (same version, both omnibus installations).
All according to Convert Community Edition to Enterprise Edition | GitLab.

The process went fine, I haven’t noticed any errors.
Did gitlab-ctl reconfigure && gitlab-ctl restart and gitlab-ctl status showed that everything is up and running, but puma and sidekiq are restarting every ~30 seconds.
gitlab-rake gitlab:check SANITIZE=true also showed that sidekiq had some problems.
Looking at the gitlab tail puma logs I found the following error:

2022-10-29_08:08:54.28274 {"timestamp":"2022-10-29T08:08:54.282Z","pid":889863,"message":"Puma starting in cluster mode..."}
2022-10-29_08:08:54.28276 {"timestamp":"2022-10-29T08:08:54.282Z","pid":889863,"message":"* Puma version: 5.6.5 (ruby 2.7.5-p203) (\"Birdie's Version\")"}
2022-10-29_08:08:54.28276 {"timestamp":"2022-10-29T08:08:54.282Z","pid":889863,"message":"*  Min threads: 4"}
2022-10-29_08:08:54.28276 {"timestamp":"2022-10-29T08:08:54.282Z","pid":889863,"message":"*  Max threads: 4"}
2022-10-29_08:08:54.28277 {"timestamp":"2022-10-29T08:08:54.282Z","pid":889863,"message":"*  Environment: production"}
2022-10-29_08:08:54.28277 {"timestamp":"2022-10-29T08:08:54.282Z","pid":889863,"message":"*   Master PID: 889863"}
2022-10-29_08:08:54.28278 {"timestamp":"2022-10-29T08:08:54.282Z","pid":889863,"message":"*      Workers: 4"}
2022-10-29_08:08:54.28278 {"timestamp":"2022-10-29T08:08:54.282Z","pid":889863,"message":"*     Restarts: (✔) hot (✖) phased"}
2022-10-29_08:08:54.28279 {"timestamp":"2022-10-29T08:08:54.282Z","pid":889863,"message":"* Preloading application"}
2022-10-29_08:09:19.75913 {"timestamp":"2022-10-29T08:09:19.758Z","pid":889863,"message":"! Unable to load application: ArgumentError: invalid byte sequence in UTF-8"}
2022-10-29_08:09:19.76022 bundler: failed to load command: puma (/opt/gitlab/embedded/bin/puma)
2022-10-29_08:09:19.76055 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/x509/certificate.rb:89:in `scan': invalid byte sequence in UTF-8 (ArgumentError)
2022-10-29_08:09:19.76056 	from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/x509/certificate.rb:89:in `load_ca_certs_bundle'
2022-10-29_08:09:19.76056 	from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/x509/certificate.rb:65:in `block (2 levels) in ca_certs_bundle'
2022-10-29_08:09:19.76057 	from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/x509/certificate.rb:64:in `each'
2022-10-29_08:09:19.76057 	from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/x509/certificate.rb:64:in `flat_map'
2022-10-29_08:09:19.76058 	from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/x509/certificate.rb:64:in `block in ca_certs_bundle'
2022-10-29_08:09:19.76058 	from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/utils/strong_memoize.rb:44:in `strong_memoize'
2022-10-29_08:09:19.76058 	from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/x509/certificate.rb:63:in `ca_certs_bundle'
2022-10-29_08:09:19.76059 	from /opt/gitlab/embedded/service/gitlab-rails/ee/lib/gitlab/applied_ml/suggested_reviewers/client.rb:11:in `<class:Client>'
2022-10-29_08:09:19.76059 	from /opt/gitlab/embedded/service/gitlab-rails/ee/lib/gitlab/applied_ml/suggested_reviewers/client.rb:6:in `<module:SuggestedReviewers>'
2022-10-29_08:09:19.76059 	from /opt/gitlab/embedded/service/gitlab-rails/ee/lib/gitlab/applied_ml/suggested_reviewers/client.rb:5:in `<module:AppliedMl>'
2022-10-29_08:09:19.76059 	from /opt/gitlab/embedded/service/gitlab-rails/ee/lib/gitlab/applied_ml/suggested_reviewers/client.rb:4:in `<module:Gitlab>'
2022-10-29_08:09:19.76060 	from /opt/gitlab/embedded/service/gitlab-rails/ee/lib/gitlab/applied_ml/suggested_reviewers/client.rb:3:in `<top (required)>'
2022-10-29_08:09:19.76060 	from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/zeitwerk-2.6.0/lib/zeitwerk/kernel.rb:27:in `require'
2022-10-29_08:09:19.76060 	from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/zeitwerk-2.6.0/lib/zeitwerk/kernel.rb:27:in `require'
2022-10-29_08:09:19.76061 	from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/zeitwerk-2.6.0/lib/zeitwerk/loader/helpers.rb:127:in `const_get'
2022-10-29_08:09:19.76061 	from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/zeitwerk-2.6.0/lib/zeitwerk/loader/helpers.rb:127:in `cget'
2022-10-29_08:09:19.76061 	from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/zeitwerk-2.6.0/lib/zeitwerk/loader.rb:239:in `block (2 levels) in eager_load'
2022-10-29_08:09:19.76061 	from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/zeitwerk-2.6.0/lib/zeitwerk/loader/helpers.rb:41:in `block in ls'
2022-10-29_08:09:19.76063 	from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/zeitwerk-2.6.0/lib/zeitwerk/loader/helpers.rb:27:in `each'
2022-10-29_08:09:19.76063 	from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/zeitwerk-2.6.0/lib/zeitwerk/loader/helpers.rb:27:in `ls'
2022-10-29_08:09:19.76063 	from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/zeitwerk-2.6.0/lib/zeitwerk/loader.rb:234:in `block in eager_load'
2022-10-29_08:09:19.76063 	from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/zeitwerk-2.6.0/lib/zeitwerk/loader.rb:219:in `synchronize'
2022-10-29_08:09:19.76063 	from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/zeitwerk-2.6.0/lib/zeitwerk/loader.rb:219:in `eager_load'
2022-10-29_08:09:19.76064 	from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/zeitwerk-2.6.0/lib/zeitwerk/loader.rb:318:in `each'
2022-10-29_08:09:19.76064 	from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/zeitwerk-2.6.0/lib/zeitwerk/loader.rb:318:in `eager_load_all'
2022-10-29_08:09:19.76064 	from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/railties-6.1.6.1/lib/rails/application/finisher.rb:133:in `block in <module:Finisher>'
2022-10-29_08:09:19.76065 	from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/railties-6.1.6.1/lib/rails/initializable.rb:32:in `instance_exec'
2022-10-29_08:09:19.76065 	from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/railties-6.1.6.1/lib/rails/initializable.rb:32:in `run'
2022-10-29_08:09:19.76065 	from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/railties-6.1.6.1/lib/rails/initializable.rb:61:in `block in run_initializers'
2022-10-29_08:09:19.76065 	from /opt/gitlab/embedded/lib/ruby/2.7.0/tsort.rb:228:in `block in tsort_each'
2022-10-29_08:09:19.76065 	from /opt/gitlab/embedded/lib/ruby/2.7.0/tsort.rb:350:in `block (2 levels) in each_strongly_connected_component'
2022-10-29_08:09:19.76065 	from /opt/gitlab/embedded/lib/ruby/2.7.0/tsort.rb:422:in `block (2 levels) in each_strongly_connected_component_from'
2022-10-29_08:09:19.76066 	from /opt/gitlab/embedded/lib/ruby/2.7.0/tsort.rb:422:in `block (2 levels) in each_strongly_connected_component_from'
2022-10-29_08:09:19.76066 	from /opt/gitlab/embedded/lib/ruby/2.7.0/tsort.rb:422:in `block (2 levels) in each_strongly_connected_component_from'
2022-10-29_08:09:19.76066 	from /opt/gitlab/embedded/lib/ruby/2.7.0/tsort.rb:422:in `block (2 levels) in each_strongly_connected_component_from'
2022-10-29_08:09:19.76067 	from /opt/gitlab/embedded/lib/ruby/2.7.0/tsort.rb:431:in `each_strongly_connected_component_from'
2022-10-29_08:09:19.76067 	from /opt/gitlab/embedded/lib/ruby/2.7.0/tsort.rb:421:in `block in each_strongly_connected_component_from'
2022-10-29_08:09:19.76067 	from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/railties-6.1.6.1/lib/rails/initializable.rb:50:in `each'
2022-10-29_08:09:19.76067 	from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/railties-6.1.6.1/lib/rails/initializable.rb:50:in `tsort_each_child'
2022-10-29_08:09:19.76067 	from /opt/gitlab/embedded/lib/ruby/2.7.0/tsort.rb:415:in `call'
2022-10-29_08:09:19.76067 	from /opt/gitlab/embedded/lib/ruby/2.7.0/tsort.rb:415:in `each_strongly_connected_component_from'
2022-10-29_08:09:19.76068 	from /opt/gitlab/embedded/lib/ruby/2.7.0/tsort.rb:421:in `block in each_strongly_connected_component_from'
2022-10-29_08:09:19.76068 	from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/railties-6.1.6.1/lib/rails/initializable.rb:50:in `each'
2022-10-29_08:09:19.76069 	from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/railties-6.1.6.1/lib/rails/initializable.rb:50:in `tsort_each_child'
2022-10-29_08:09:19.76069 	from /opt/gitlab/embedded/lib/ruby/2.7.0/tsort.rb:415:in `call'
2022-10-29_08:09:19.76069 	from /opt/gitlab/embedded/lib/ruby/2.7.0/tsort.rb:415:in `each_strongly_connected_component_from'
2022-10-29_08:09:19.76069 	from /opt/gitlab/embedded/lib/ruby/2.7.0/tsort.rb:421:in `block in each_strongly_connected_component_from'
2022-10-29_08:09:19.76069 	from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/railties-6.1.6.1/lib/rails/initializable.rb:50:in `each'
2022-10-29_08:09:19.76070 	from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/railties-6.1.6.1/lib/rails/initializable.rb:50:in `tsort_each_child'
2022-10-29_08:09:19.76070 	from /opt/gitlab/embedded/lib/ruby/2.7.0/tsort.rb:415:in `call'
2022-10-29_08:09:19.76070 	from /opt/gitlab/embedded/lib/ruby/2.7.0/tsort.rb:415:in `each_strongly_connected_component_from'
2022-10-29_08:09:19.76070 	from /opt/gitlab/embedded/lib/ruby/2.7.0/tsort.rb:421:in `block in each_strongly_connected_component_from'
2022-10-29_08:09:19.76071 	from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/railties-6.1.6.1/lib/rails/initializable.rb:50:in `each'
2022-10-29_08:09:19.76071 	from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/railties-6.1.6.1/lib/rails/initializable.rb:50:in `tsort_each_child'
2022-10-29_08:09:19.76071 	from /opt/gitlab/embedded/lib/ruby/2.7.0/tsort.rb:415:in `call'
2022-10-29_08:09:19.76071 	from /opt/gitlab/embedded/lib/ruby/2.7.0/tsort.rb:415:in `each_strongly_connected_component_from'
2022-10-29_08:09:19.76071 	from /opt/gitlab/embedded/lib/ruby/2.7.0/tsort.rb:349:in `block in each_strongly_connected_component'
2022-10-29_08:09:19.76072 	from /opt/gitlab/embedded/lib/ruby/2.7.0/tsort.rb:347:in `each'
2022-10-29_08:09:19.76072 	from /opt/gitlab/embedded/lib/ruby/2.7.0/tsort.rb:347:in `call'
2022-10-29_08:09:19.76072 	from /opt/gitlab/embedded/lib/ruby/2.7.0/tsort.rb:347:in `each_strongly_connected_component'
2022-10-29_08:09:19.76072 	from /opt/gitlab/embedded/lib/ruby/2.7.0/tsort.rb:226:in `tsort_each'
2022-10-29_08:09:19.76072 	from /opt/gitlab/embedded/lib/ruby/2.7.0/tsort.rb:205:in `tsort_each'
2022-10-29_08:09:19.76072 	from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/railties-6.1.6.1/lib/rails/initializable.rb:60:in `run_initializers'
2022-10-29_08:09:19.76073 	from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/railties-6.1.6.1/lib/rails/application.rb:391:in `initialize!'
2022-10-29_08:09:19.76073 	from /opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:7:in `<top (required)>'
2022-10-29_08:09:19.76074 	from /opt/gitlab/embedded/service/gitlab-rails/config.ru:5:in `require'
2022-10-29_08:09:19.76074 	from /opt/gitlab/embedded/service/gitlab-rails/config.ru:5:in `block in <main>'
2022-10-29_08:09:19.76074 	from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rack-2.2.4/lib/rack/builder.rb:116:in `eval'
2022-10-29_08:09:19.76074 	from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rack-2.2.4/lib/rack/builder.rb:116:in `new_from_string'
2022-10-29_08:09:19.76074 	from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rack-2.2.4/lib/rack/builder.rb:105:in `load_file'
2022-10-29_08:09:19.76074 	from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rack-2.2.4/lib/rack/builder.rb:66:in `parse_file'
2022-10-29_08:09:19.76075 	from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/puma-5.6.5/lib/puma/configuration.rb:348:in `load_rackup'
2022-10-29_08:09:19.76075 	from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/puma-5.6.5/lib/puma/configuration.rb:270:in `app'
2022-10-29_08:09:19.76075 	from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/puma-5.6.5/lib/puma/runner.rb:150:in `load_and_bind'
2022-10-29_08:09:19.76076 	from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/puma-5.6.5/lib/puma/cluster.rb:357:in `run'
2022-10-29_08:09:19.76076 	from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/puma-5.6.5/lib/puma/launcher.rb:193:in `run'
2022-10-29_08:09:19.76076 	from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/puma-5.6.5/lib/puma/cli.rb:81:in `run'
2022-10-29_08:09:19.76076 	from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/puma-5.6.5/bin/puma:10:in `<top (required)>'
2022-10-29_08:09:19.76076 	from /opt/gitlab/embedded/bin/puma:23:in `load'
2022-10-29_08:09:19.76076 	from /opt/gitlab/embedded/bin/puma:23:in `<top (required)>'
2022-10-29_08:09:19.76076 	from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/bundler-2.3.15/lib/bundler/cli/exec.rb:58:in `load'
2022-10-29_08:09:19.76077 	from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/bundler-2.3.15/lib/bundler/cli/exec.rb:58:in `kernel_load'
2022-10-29_08:09:19.76077 	from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/bundler-2.3.15/lib/bundler/cli/exec.rb:23:in `run'
2022-10-29_08:09:19.76077 	from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/bundler-2.3.15/lib/bundler/cli.rb:483:in `exec'
2022-10-29_08:09:19.76077 	from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/bundler-2.3.15/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
2022-10-29_08:09:19.76078 	from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/bundler-2.3.15/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
2022-10-29_08:09:19.76078 	from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/bundler-2.3.15/lib/bundler/vendor/thor/lib/thor.rb:392:in `dispatch'
2022-10-29_08:09:19.76078 	from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/bundler-2.3.15/lib/bundler/cli.rb:31:in `dispatch'
2022-10-29_08:09:19.76079 	from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/bundler-2.3.15/lib/bundler/vendor/thor/lib/thor/base.rb:485:in `start'
2022-10-29_08:09:19.76079 	from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/bundler-2.3.15/lib/bundler/cli.rb:25:in `start'
2022-10-29_08:09:19.76079 	from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/bundler-2.3.15/exe/bundle:48:in `block in <top (required)>'
2022-10-29_08:09:19.76079 	from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/bundler-2.3.15/lib/bundler/friendly_errors.rb:117:in `with_friendly_errors'
2022-10-29_08:09:19.76079 	from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/bundler-2.3.15/exe/bundle:36:in `<top (required)>'
2022-10-29_08:09:19.76080 	from /opt/gitlab/embedded/bin/bundle:23:in `load'
2022-10-29_08:09:19.76080 	from /opt/gitlab/embedded/bin/bundle:23:in `<main>'

So, some problem with encoding in certificates?! :confused:

I then looked at /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/x509/certificate.rb and commented out the following code from the self.load_ca_certs_bundle(ca_certs_string) function:

#ca_certs_string.scan(CERT_REGEX).map do |ca_cert_string|
#  OpenSSL::X509::Certificate.new(ca_cert_string)
#end

After gitlab-ctl reconfigure && gitlab-ctl restart puma successfuly restarted and gitlab now works fine.

Our installation is behind reverse-proxy where we terminate certificates. We don’t have any certificates issued with gitlab and specified in /etc/gitlab/gitlab.rb, but I’m still not sure if commenting the lines above is safe.

The questions are: why I got this error in the first place, how can I fix it and is it safe to keep the lines above commented out until I find the root cause?

Thanks a lot for any help! :slight_smile:

Solved.
There was some weird certificate in /etc/giltab/trusted-certs. Removing that certificate fixed the issue.