Upgrade to 16.3.0-ee.0 failing

sram · August 23, 2023, 2:34am

Our gitlab instance runs on Ubuntu 20.04 and is configured to update automatically. We also have a geo secondary node configured, which also encountered the same issue.

Today morning it tried updating to 16.3.0 and failed with the below error.

APT error logs

[2023-08-23T02:31:52+00:00] FATAL: Stacktrace dumped to /opt/gitlab/embedded/cookbooks/cache/cinc-stacktrace.out
[2023-08-23T02:31:52+00:00] FATAL: ---------------------------------------------------------------------------------------
[2023-08-23T02:31:52+00:00] FATAL: PLEASE PROVIDE THE CONTENTS OF THE stacktrace.out FILE (above) IF YOU FILE A BUG REPORT
[2023-08-23T02:31:52+00:00] FATAL: ---------------------------------------------------------------------------------------
[2023-08-23T02:31:52+00:00] FATAL: Mixlib::ShellOut::ShellCommandFailed: rails_migration[gitlab-rails] (gitlab::database_migrations line 51) had an error: Mixlib::ShellOut::ShellCommandFailed: bash_hide_env[migrate gitlab-rails database] (gitlab::database_migrations line 20) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received ‘1’
---- Begin output of “bash” ----
STDOUT: rake aborted!
ArgumentError: is not a recognized provider
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/content_security_policy/config_loader.rb:197:in `build_lfs_url' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/content_security_policy/config_loader.rb:89:in` allow_lfs’
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/content_security_policy/config_loader.rb:24:in `default_directives' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/content_security_policy/config_loader.rb:206:in` initialize’
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/content_security_policy.rb:12:in `new' /opt/gitlab/embedded/service/gitlab-rails/config/initializers/content_security_policy.rb:12:in` block in <top (required)>’
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/content_security_policy.rb:11:in `<top (required)>' /opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:7:in` <top (required)>’
internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.0.0/rubygems/core_ext/kernel_require.rb:38:in `require' <internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.0.0/rubygems/core_ext/kernel_require.rb>:38:in` require’
/opt/gitlab/embedded/bin/bundle:25:in `load' /opt/gitlab/embedded/bin/bundle:25:in` ’
Tasks: TOP => gitlab:db:configure => environment
(See full trace by running task with --trace)
STDERR:
---- End output of “bash” ----
Ran “bash” returned 1

There was an error running gitlab-ctl reconfigure. Please check the output above for more
details.

Here is the stacktrace

Generated at 2023-08-23 02:31:52 +0000
Mixlib::ShellOut::ShellCommandFailed: rails_migration[gitlab-rails] (gitlab::database_migrations line 51) had an error: Mixlib::ShellOut::ShellCommandFailed: bash_hide_env[migrate gitlab-rails database] (gitlab::database_migrations line 20) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received ‘1’
---- Begin output of “bash” ----
STDOUT: rake aborted!
ArgumentError: is not a recognized provider
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/content_security_policy/config_loader.rb:197:in build_lfs_url' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/content_security_policy/config_loader.rb:89:in allow_lfs’
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/content_security_policy/config_loader.rb:24:in default_directives' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/content_security_policy/config_loader.rb:206:in initialize’
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/content_security_policy.rb:12:in new' /opt/gitlab/embedded/service/gitlab-rails/config/initializers/content_security_policy.rb:12:in block in <top (required)>’
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/content_security_policy.rb:11:in <top (required)>' /opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:7:in <top (required)>’
internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.0.0/rubygems/core_ext/kernel_require.rb:38:in require' <internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.0.0/rubygems/core_ext/kernel_require.rb>:38:in require’
/opt/gitlab/embedded/bin/bundle:25:in load' /opt/gitlab/embedded/bin/bundle:25:in ’
Tasks: TOP => gitlab:db:configure => environment
(See full trace by running task with --trace)
STDERR:
---- End output of “bash” ----
Ran “bash” returned 1
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/mixlib-shellout-3.2.7/lib/mixlib/shellout.rb:300:in invalid!' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/mixlib-shellout-3.2.7/lib/mixlib/shellout.rb:287:in error!’
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/mixlib-shellout-3.2.7/lib/mixlib/shellout/helper.rb:130:in shell_out_compacted!' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/mixlib-shellout-3.2.7/lib/mixlib/shellout/helper.rb:54:in shell_out!’
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/provider/execute.rb:52:in block (2 levels) in <class:Execute>' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/mixin/why_run.rb:51:in add_action’
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/provider.rb:293:in converge_by' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/provider/execute.rb:50:in block in class:Execute’
(eval):2:in block in action_run' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/provider.rb:304:in instance_eval’
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/provider.rb:304:in compile_and_converge_action' (eval):2:in action_run’
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/provider.rb:245:in run_action' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/resource.rb:601:in block in run_action’
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/resource.rb:628:in with_umask' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/resource.rb:600:in run_action’
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/runner.rb:74:in run_action' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/runner.rb:108:in block in run_all_actions’
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/runner.rb:108:in each' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/runner.rb:108:in run_all_actions’
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/resource_collection.rb:64:in insert' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/dsl/declare_resource.rb:267:in declare_resource’
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/dsl/resources.rb:36:in bash_hide_env' /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb:20:in block in class_from_file’
(eval):2:in block in action_run' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/provider.rb:304:in instance_eval’
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/provider.rb:304:in compile_and_converge_action' (eval):2:in action_run’
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/provider.rb:245:in run_action' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/resource.rb:601:in block in run_action’
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/resource.rb:628:in with_umask' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/resource.rb:600:in run_action’
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/runner.rb:74:in run_action' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/runner.rb:108:in block in run_all_actions’
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/runner.rb:108:in each' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/runner.rb:108:in run_all_actions’
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/runner.rb:132:in block in converge' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/resource_collection/resource_list.rb:96:in block in execute_each_resource’
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/resource_collection/stepable_iterator.rb:114:in call_iterator_block' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/resource_collection/stepable_iterator.rb:85:in step’
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/resource_collection/stepable_iterator.rb:103:in iterate' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/resource_collection/stepable_iterator.rb:54:in each_with_index’
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/resource_collection/resource_list.rb:94:in execute_each_resource' /opt/gitlab/embedded/lib/ruby/3.0.0/forwardable.rb:238:in execute_each_resource’
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/runner.rb:130:in converge' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/client.rb:692:in block in converge’
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/client.rb:687:in catch' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/client.rb:687:in converge’
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/client.rb:711:in converge_and_save' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/client.rb:285:in run’
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/application.rb:305:in run_with_graceful_exit_option' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/application.rb:281:in block in run_chef_client’
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/local_mode.rb:42:in with_server_connectivity' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/application.rb:264:in run_chef_client’
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/application/base.rb:352:in run_application' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/application.rb:67:in run’
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-bin-17.10.0/bin/cinc-client:25:in <top (required)>' /opt/gitlab/embedded/bin/cinc-client:25:in load’
/opt/gitlab/embedded/bin/cinc-client:25:in `’

Caused by Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received ‘1’
---- Begin output of “bash” ----
STDOUT: rake aborted!
ArgumentError: is not a recognized provider
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/content_security_policy/config_loader.rb:197:in build_lfs_url' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/content_security_policy/config_loader.rb:89:in allow_lfs’
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/content_security_policy/config_loader.rb:24:in default_directives' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/content_security_policy/config_loader.rb:206:in initialize’
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/content_security_policy.rb:12:in new' /opt/gitlab/embedded/service/gitlab-rails/config/initializers/content_security_policy.rb:12:in block in <top (required)>’
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/content_security_policy.rb:11:in <top (required)>' /opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:7:in <top (required)>’
internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.0.0/rubygems/core_ext/kernel_require.rb:38:in require' <internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.0.0/rubygems/core_ext/kernel_require.rb>:38:in require’
/opt/gitlab/embedded/bin/bundle:25:in load' /opt/gitlab/embedded/bin/bundle:25:in ’
Tasks: TOP => gitlab:db:configure => environment
(See full trace by running task with --trace)
STDERR:
---- End output of “bash” ----
Ran “bash” returned 1
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/mixlib-shellout-3.2.7/lib/mixlib/shellout.rb:300:in invalid!' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/mixlib-shellout-3.2.7/lib/mixlib/shellout.rb:287:in error!’
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/mixlib-shellout-3.2.7/lib/mixlib/shellout/helper.rb:130:in shell_out_compacted!' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/mixlib-shellout-3.2.7/lib/mixlib/shellout/helper.rb:54:in shell_out!’
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/provider/execute.rb:52:in block (2 levels) in <class:Execute>' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/mixin/why_run.rb:51:in add_action’
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/provider.rb:293:in converge_by' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/provider/execute.rb:50:in block in class:Execute’
(eval):2:in block in action_run' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/provider.rb:304:in instance_eval’
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/provider.rb:304:in compile_and_converge_action' (eval):2:in action_run’
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/provider.rb:245:in run_action' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/resource.rb:601:in block in run_action’
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/resource.rb:628:in with_umask' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/resource.rb:600:in run_action’
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/runner.rb:74:in run_action' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/runner.rb:108:in block in run_all_actions’
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/runner.rb:108:in each' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/runner.rb:108:in run_all_actions’
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/resource_collection.rb:64:in insert' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/dsl/declare_resource.rb:267:in declare_resource’
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/dsl/resources.rb:36:in bash_hide_env' /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb:20:in block in class_from_file’
(eval):2:in block in action_run' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/provider.rb:304:in instance_eval’
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/provider.rb:304:in compile_and_converge_action' (eval):2:in action_run’
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/provider.rb:245:in run_action' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/resource.rb:601:in block in run_action’
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/resource.rb:628:in with_umask' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/resource.rb:600:in run_action’
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/runner.rb:74:in run_action' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/runner.rb:108:in block in run_all_actions’
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/runner.rb:108:in each' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/runner.rb:108:in run_all_actions’
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/runner.rb:132:in block in converge' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/resource_collection/resource_list.rb:96:in block in execute_each_resource’
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/resource_collection/stepable_iterator.rb:114:in call_iterator_block' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/resource_collection/stepable_iterator.rb:85:in step’
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/resource_collection/stepable_iterator.rb:103:in iterate' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/resource_collection/stepable_iterator.rb:54:in each_with_index’
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/resource_collection/resource_list.rb:94:in execute_each_resource' /opt/gitlab/embedded/lib/ruby/3.0.0/forwardable.rb:238:in execute_each_resource’
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/runner.rb:130:in converge' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/client.rb:692:in block in converge’
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/client.rb:687:in catch' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/client.rb:687:in converge’
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/client.rb:711:in converge_and_save' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/client.rb:285:in run’
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/application.rb:305:in run_with_graceful_exit_option' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/application.rb:281:in block in run_chef_client’
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/local_mode.rb:42:in with_server_connectivity' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/application.rb:264:in run_chef_client’
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/application/base.rb:352:in run_application' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/application.rb:67:in run’
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-bin-17.10.0/bin/cinc-client:25:in <top (required)>' /opt/gitlab/embedded/bin/cinc-client:25:in load’
/opt/gitlab/embedded/bin/cinc-client:25:in `’

Any help is appriciated

sram · August 23, 2023, 3:17am

It seems it was our CSP configs that was causing the issues. Triggering reconfigure after disabling it in gitlab.rb fixed the issue.

n0r · August 23, 2023, 8:26am

We were facing the same issue where disabling CSP helped. After successfully reconfiguring (which triggered many db migrations) we tried reactivating CSP → error was back.

Our CSP:

gitlab_rails['content_security_policy'] = {
  enabled: true,
  report_only: false,
  directives: {
    frame_ancestors: "FQDN redacted"
  }
}

gitlab-ctl reconfigure with activated CSP error:

rails_migration[gitlab-rails] (gitlab::database_migrations line 51) had an error: Mixlib::ShellOut::ShellCommandFailed: bash_hide_env[migrate gitlab-rails database] (gitlab::database_migrations line 20) had an error: Mixlib::ShellOut::ShellCommandFailed:

And:

ArgumentError:  is not a recognized provider                                                                                                                   
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/content_security_policy/config_loader.rb:197:in `build_lfs_url'                                           
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/content_security_policy/config_loader.rb:89:in `allow_lfs'                                                
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/content_security_policy/config_loader.rb:24:in `default_directives'                                       
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/content_security_policy/config_loader.rb:206:in `initialize'                                              
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/content_security_policy.rb:12:in `new'                                                           
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/content_security_policy.rb:12:in `block in <top (required)>'                                     
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/content_security_policy.rb:11:in `<top (required)>'                                              
/opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:7:in `<top (required)>'                                                                        
<internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.0.0/rubygems/core_ext/kernel_require.rb>:38:in `require'                                                   
<internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.0.0/rubygems/core_ext/kernel_require.rb>:38:in `require'                                                   
/opt/gitlab/embedded/bin/bundle:25:in `load'                                                                                                                   
/opt/gitlab/embedded/bin/bundle:25:in `<main>'                                                                                                                 
Tasks: TOP => cache:clear => cache:clear:redis => environment                                                                                                  
(See full trace by running task with --trace)                                                                                                                  
---- End output of /opt/gitlab/bin/gitlab-rake cache:clear ----                                                                                                
Ran /opt/gitlab/bin/gitlab-rake cache:clear returned 1

pilou · August 23, 2023, 8:32am

Related issue : With 16.3.0 reconfigure fails when ContentSecurityPolicy is set and LFS objects are on local disk (#422998) · Issues · GitLab.org / GitLab · GitLab

themthem58 · August 23, 2023, 10:21am

So the solution is desactivate LFS or desactivate CSP settings ? ^^’ It’s not really a good solution …

dnsmichi · August 23, 2023, 2:50pm

Thanks for sharing. The issue was closed as duplicate, suggest subscribing to the new issue Puma ArgumentError: is not a recognized provider (#422936) · Issues · GitLab.org / GitLab · GitLab

iwalker · August 23, 2023, 3:06pm

Even with the default csp config it fails to upgrade:

# Content Security Policy
gitlab_rails['content_security_policy'] = {
 'enabled' => true,
}

so disabled it by setting to false. Default LFS path is in use.

I’m not even using object storage.

Upgrade to 16.3.0-ee.0 failing

There was an error running gitlab-ctl reconfigure. Please check the output above for more details.

There was an error running gitlab-ctl reconfigure. Please check the output above for more
details.