Upgrading omnibus from 15.0.5 to 15.1.6 on Aliyun ECS instance throws warning making gitlab-ctl slow to complete

andrewwai · June 7, 2023, 3:30am

Hi I have been following the upgrade guide from https://docs.gitlab.com/ee/update and have hit a platform specific issue when trying to pass version 15.1.6. I am running on an Alibabacloud ECS instance with 2 cores and 8gb ram. When I get to 15.1.6, gitlab-ctl reconfigure takes 15 minutes and logs show:

[2023-06-06T08:53:42+00:00] INFO: Started Cinc Zero at chefzero://localhost:1 with repository at /opt/gitlab/embedded (One version per cookbook)
[2023-06-06T08:53:42+00:00] INFO: *** Cinc Client 17.10.0 ***
[2023-06-06T08:53:42+00:00] INFO: Platform: x86_64-linux
[2023-06-06T08:53:42+00:00] INFO: Cinc-client pid: 25
[2023-06-06T09:08:35+00:00] WARN: Plugin Alibaba threw exception #<SystemStackError: stack level too deep> /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/net-
protocol-0.1.3/lib/net/protocol.rb:279:in `block in write0'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/net-protocol-0.1.3/lib/net/protocol.rb:277:in `each'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/net-protocol-0.1.3/lib/net/protocol.rb:277:in `each_with_index'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/net-protocol-0.1.3/lib/net/protocol.rb:277:in `write0'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/net-protocol-0.1.3/lib/net/protocol.rb:250:in `block in write'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/net-protocol-0.1.3/lib/net/protocol.rb:267:in `writing'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/net-protocol-0.1.3/lib/net/protocol.rb:249:in `write'
/opt/gitlab/embedded/lib/ruby/3.0.0/net/http/generic_request.rb:335:in `write_header'
/opt/gitlab/embedded/lib/ruby/3.0.0/net/http/generic_request.rb:129:in `exec'
/opt/gitlab/embedded/lib/ruby/3.0.0/net/http.rb:1550:in `block in transport_request'
/opt/gitlab/embedded/lib/ruby/3.0.0/net/http.rb:1548:in `catch'
/opt/gitlab/embedded/lib/ruby/3.0.0/net/http.rb:1548:in `transport_request'
/opt/gitlab/embedded/lib/ruby/3.0.0/net/http.rb:1521:in `request'
/opt/gitlab/embedded/lib/ruby/3.0.0/net/http.rb:1270:in `get'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/ohai-17.9.0/lib/ohai/mixin/alibaba_metadata.rb:38:in `http_get'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/ohai-17.9.0/lib/ohai/mixin/alibaba_metadata.rb:42:in `fetch_metadata'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/ohai-17.9.0/lib/ohai/mixin/alibaba_metadata.rb:52:in `block in fetch_metadata'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/ohai-17.9.0/lib/ohai/mixin/alibaba_metadata.rb:51:in `each'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/ohai-17.9.0/lib/ohai/mixin/alibaba_metadata.rb:51:in `fetch_metadata'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/ohai-17.9.0/lib/ohai/mixin/alibaba_metadata.rb:52:in `block in fetch_metadata'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/ohai-17.9.0/lib/ohai/mixin/alibaba_metadata.rb:51:in `each'
.
(repeats and removed for clarity)
.
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/ohai-17.9.0/lib/ohai/mixin/alibaba_metadata.rb:51:in `fetch_metadata'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/ohai-17.9.0/lib/ohai/plugins/alibaba.rb:64:in `block (2 levels) in <main>'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/ohai-17.9.0/lib/ohai/dsl/plugin/versionvii.rb:139:in `instance_eval'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/ohai-17.9.0/lib/ohai/dsl/plugin/versionvii.rb:139:in `run_plugin'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/ohai-17.9.0/lib/ohai/dsl/plugin.rb:115:in `run'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/ohai-17.9.0/lib/ohai/dsl/plugin.rb:200:in `safe_run'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/ohai-17.9.0/lib/ohai/runner.rb:99:in `run_v7_plugin'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/ohai-17.9.0/lib/ohai/runner.rb:57:in `block in run_plugin'
/opt/gitlab/embedded/lib/ruby/3.0.0/benchmark.rb:308:in `realtime'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/ohai-17.9.0/lib/ohai/runner.rb:50:in `run_plugin'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/ohai-17.9.0/lib/ohai/system.rb:131:in `block in run_plugins'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/ohai-17.9.0/lib/ohai/system.rb:130:in `each'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/ohai-17.9.0/lib/ohai/system.rb:130:in `run_plugins'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/ohai-17.9.0/lib/ohai/system.rb:105:in `all_plugins'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/client.rb:595:in `run_ohai'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/client.rb:256:in `run'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/application.rb:305:in `run_with_graceful_exit_option'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/application.rb:281:in `block in run_chef_client'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/local_mode.rb:42:in `with_server_connectivity'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/application.rb:264:in `run_chef_client'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/application/base.rb:352:in `run_application'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/application.rb:67:in `run'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-bin-17.10.0/bin/cinc-client:25:in `<top (required)>'
/opt/gitlab/embedded/bin/cinc-client:25:in `load'
/opt/gitlab/embedded/bin/cinc-client:25:in `<main>'
[2023-06-06T09:08:36+00:00] INFO: Setting the run_list to ["recipe[gitlab]"] from CLI options
[2023-06-06T09:08:36+00:00] INFO: Run List is [recipe[gitlab]]

Gitlab-ctl reconfigure eventually completes and gitlab runs fine but just seems to take a long time. Any pointers on how to configure the Alibaba Plugin?

I have tried with Centos7, Ubuntu and even Aliyun Linux both via host package installs and in docker, all produce the same issue.

Thanks

Update:
For anyone facing similar issues with gitlab-ctl reconfigure the fix was upgrading the instance to 4 cores. Thank you to anyway who may have looked into this.

andrewwai · June 9, 2023, 2:18am

Update: Spoke too soon. The same issue appears once I restore the repository and proceed with the minor upgrade from 15.0.5 to 15.1.6. Some additional information. When upgrading from 14.10.5 to 15.0.5 the gitbal-ctl reconfigure runs and completes without the issue. It only appears upgrading from 15.0.5 onwards.

sagehou · July 24, 2023, 2:34am

I encountered the same problem! My ECS instance has allocated 4 cores and 8gb ram.
I have tried centos7 and docker, upgrading process always failed and gitlab-ctl reconfigure will threw warnings after 10 minutes.

Is there any other way to solve this problem? Every time the update is scary!

andrewwai · August 2, 2023, 10:13am

Hi, have you resolved your problem? I retried some time later by replicating an ecs instance in centos 7. When upgrading be sure to let the background migrations complete before proceeding to the next incremental upgrade. Hope that helps.