Gitlab-rake fails with "ArgumentError: wrong number of arguments (given 2, expected 0)"

In the course of doing a yum update on my home GitLab server (which upgraded from from CentOS 7.6 to 7.7, although I hope that is unrelated), I noticed that the gitlab-ce update (12.2.4-ce.0.el7.x86_64 -> 12.2.5-ce.0.el7.x86_64) failed.

Long story short, most gitlab-rake commands fail like this:

# gitlab-rake gitlab:check
rake aborted!
ArgumentError: wrong number of arguments (given 2, expected 0)
(eval):1:in `store'
/opt/gitlab/embedded/service/gitlab-rails/lib/banzai/filter/asset_proxy_filter.rb:41:in `initialize_settings'
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/asset_proxy_settings.rb:5:in `block in <top (required)>'
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/asset_proxy_settings.rb:4:in `<top (required)>'
/opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:6:in `<top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => gitlab:check => gitlab_environment => environment
(See full trace by running task with --trace)

Line 41 of asset_proxy_filter.rb:

Gitlab.config['asset_proxy'] ||= Settingslogic.new({})

I’m not sure how to debug this further.

1 Like

Full trace:

# gitlab-rake -t gitlab:check
** Invoke gitlab:check (first_time)
** Invoke gitlab_environment (first_time)
** Invoke environment (first_time)
** Execute environment
rake aborted!
ArgumentError: wrong number of arguments (given 2, expected 0)
(eval):1:in `store'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/settingslogic-2.0.9/lib/settingslogic.rb:43:in `[]='
/opt/gitlab/embedded/service/gitlab-rails/lib/banzai/filter/asset_proxy_filter.rb:41:in `initialize_settings'
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/asset_proxy_settings.rb:5:in `block in <top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-5.2.3/lib/active_support/lazy_load_hooks.rb:71:in `instance_eval'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-5.2.3/lib/active_support/lazy_load_hooks.rb:71:in `block in execute_hook'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-5.2.3/lib/active_support/lazy_load_hooks.rb:62:in `with_execution_control'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-5.2.3/lib/active_support/lazy_load_hooks.rb:67:in `execute_hook'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-5.2.3/lib/active_support/lazy_load_hooks.rb:43:in `block in on_load'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-5.2.3/lib/active_support/lazy_load_hooks.rb:42:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-5.2.3/lib/active_support/lazy_load_hooks.rb:42:in `on_load'
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/asset_proxy_settings.rb:4:in `<top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-5.2.3/lib/active_support/dependencies.rb:285:in `load'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-5.2.3/lib/active_support/dependencies.rb:285:in `block in load'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-5.2.3/lib/active_support/dependencies.rb:257:in `load_dependency'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-5.2.3/lib/active_support/dependencies.rb:285:in `load'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-5.2.3/lib/rails/engine.rb:657:in `block in load_config_initializer'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-5.2.3/lib/active_support/notifications.rb:170:in `instrument'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-5.2.3/lib/rails/engine.rb:656:in `load_config_initializer'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-5.2.3/lib/rails/engine.rb:614:in `block (2 levels) in <class:Engine>'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-5.2.3/lib/rails/engine.rb:613:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-5.2.3/lib/rails/engine.rb:613:in `block in <class:Engine>'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-5.2.3/lib/rails/initializable.rb:32:in `instance_exec'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-5.2.3/lib/rails/initializable.rb:32:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-5.2.3/lib/rails/initializable.rb:61:in `block in run_initializers'
/opt/gitlab/embedded/lib/ruby/2.6.0/tsort.rb:228:in `block in tsort_each'
/opt/gitlab/embedded/lib/ruby/2.6.0/tsort.rb:350:in `block (2 levels) in each_strongly_connected_component'
/opt/gitlab/embedded/lib/ruby/2.6.0/tsort.rb:422:in `block (2 levels) in each_strongly_connected_component_from'
/opt/gitlab/embedded/lib/ruby/2.6.0/tsort.rb:431:in `each_strongly_connected_component_from'
/opt/gitlab/embedded/lib/ruby/2.6.0/tsort.rb:421:in `block in each_strongly_connected_component_from'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-5.2.3/lib/rails/initializable.rb:50:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-5.2.3/lib/rails/initializable.rb:50:in `tsort_each_child'
/opt/gitlab/embedded/lib/ruby/2.6.0/tsort.rb:415:in `call'
/opt/gitlab/embedded/lib/ruby/2.6.0/tsort.rb:415:in `each_strongly_connected_component_from'
/opt/gitlab/embedded/lib/ruby/2.6.0/tsort.rb:349:in `block in each_strongly_connected_component'
/opt/gitlab/embedded/lib/ruby/2.6.0/tsort.rb:347:in `each'
/opt/gitlab/embedded/lib/ruby/2.6.0/tsort.rb:347:in `call'
/opt/gitlab/embedded/lib/ruby/2.6.0/tsort.rb:347:in `each_strongly_connected_component'
/opt/gitlab/embedded/lib/ruby/2.6.0/tsort.rb:226:in `tsort_each'
/opt/gitlab/embedded/lib/ruby/2.6.0/tsort.rb:205:in `tsort_each'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-5.2.3/lib/rails/initializable.rb:60:in `run_initializers'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-5.2.3/lib/rails/application.rb:361:in `initialize!'
/opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:6:in `<top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-5.2.3/lib/active_support/dependencies.rb:291:in `require'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-5.2.3/lib/active_support/dependencies.rb:291:in `block in require'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-5.2.3/lib/active_support/dependencies.rb:257:in `load_dependency'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-5.2.3/lib/active_support/dependencies.rb:291:in `require'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-5.2.3/lib/rails/application.rb:337:in `require_environment!'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-5.2.3/lib/rails/application.rb:520:in `block in run_tasks_blocks'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.2/lib/rake/task.rb:273:in `block in execute'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.2/lib/rake/task.rb:273:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.2/lib/rake/task.rb:273:in `execute'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.2/lib/rake/task.rb:214:in `block in invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/2.6.0/monitor.rb:230:in `mon_synchronize'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.2/lib/rake/task.rb:194:in `invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.2/lib/rake/task.rb:238:in `block in invoke_prerequisites'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.2/lib/rake/task.rb:236:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.2/lib/rake/task.rb:236:in `invoke_prerequisites'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.2/lib/rake/task.rb:213:in `block in invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/2.6.0/monitor.rb:230:in `mon_synchronize'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.2/lib/rake/task.rb:194:in `invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.2/lib/rake/task.rb:238:in `block in invoke_prerequisites'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.2/lib/rake/task.rb:236:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.2/lib/rake/task.rb:236:in `invoke_prerequisites'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.2/lib/rake/task.rb:213:in `block in invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/2.6.0/monitor.rb:230:in `mon_synchronize'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.2/lib/rake/task.rb:194:in `invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.2/lib/rake/task.rb:183:in `invoke'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.2/lib/rake/application.rb:160:in `invoke_task'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.2/lib/rake/application.rb:116:in `block (2 levels) in top_level'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.2/lib/rake/application.rb:116:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.2/lib/rake/application.rb:116:in `block in top_level'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.2/lib/rake/application.rb:125:in `run_with_threads'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.2/lib/rake/application.rb:110:in `top_level'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.2/lib/rake/application.rb:83:in `block in run'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.2/lib/rake/application.rb:186:in `standard_exception_handling'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.2/lib/rake/application.rb:80:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.2/exe/rake:27:in `<top (required)>'
/opt/gitlab/embedded/bin/rake:23:in `load'
/opt/gitlab/embedded/bin/rake:23:in `<top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli/exec.rb:74:in `load'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli/exec.rb:74:in `kernel_load'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli/exec.rb:28:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli.rb:463:in `exec'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor/invocation.rb:126:in `invoke_command'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor.rb:387:in `dispatch'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli.rb:27:in `dispatch'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor/base.rb:466:in `start'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli.rb:18:in `start'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/exe/bundle:30:in `block in <top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/friendly_errors.rb:124:in `with_friendly_errors'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/exe/bundle:22:in `<top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => gitlab:check => gitlab_environment => environment

Hi @steven.pritchard. Welcome to the GitLab Community forum!

It sounds to me like something happened during the OS and/or GitLab upgrade that broke this, although I’m not sure what it would be. If components of GitLab went down or unresponsive after the 12.2.5 update, this could also cause problems with the rake tasks.

In many situations, a GitLab reconfigure can automatically repair or fix underlying problems.

I suggest running:
gitlab-ctl reconfigure
gitlab-ctl restart

If this doesn’t help, it might be helpful to reinstall GitLab.
yum reinstall gitlab-ee
Don’t worry, this will leave all your data intact.

Please let us know how it goes, and if you have any further questions or issues.

Sorry, I skipped over a whole bunch of my troubleshooting steps, including trying gitlab-ctl reconfigure and reinstalling the gitlab-ce package.

Part-way through gitlab-ctl reconfigure, it fails with an error caused by a failed call to gitlab-rake.

Recipe: gitlab::database_migrations
  * bash[migrate gitlab-rails database] action run
    [execute] rake aborted!
              ArgumentError: wrong number of arguments (given 2, expected 0)
              (eval):1:in `store'
              /opt/gitlab/embedded/service/gitlab-rails/lib/banzai/filter/asset_proxy_filter.rb:41:in `initialize_settings'
              /opt/gitlab/embedded/service/gitlab-rails/config/initializers/asset_proxy_settings.rb:5:in `block in <top (required)>'
              /opt/gitlab/embedded/service/gitlab-rails/config/initializers/asset_proxy_settings.rb:4:in `<top (required)>'
              /opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:6:in `<top (required)>'
              /opt/gitlab/embedded/bin/bundle:23:in `load'
              /opt/gitlab/embedded/bin/bundle:23:in `<main>'
              Tasks: TOP => gitlab:db:configure => environment
              (See full trace by running task with --trace)
    
    ================================================================================
    Error executing action `run` on resource 'bash[migrate gitlab-rails database]'
    ================================================================================
    
    Mixlib::ShellOut::ShellCommandFailed
    ------------------------------------
    Expected process to exit with [0], but received '1'
    ---- Begin output of "bash"  "/tmp/chef-script20190919-19942-dd06t1" ----
    STDOUT: rake aborted!
    ArgumentError: wrong number of arguments (given 2, expected 0)
    (eval):1:in `store'
    /opt/gitlab/embedded/service/gitlab-rails/lib/banzai/filter/asset_proxy_filter.rb:41:in `initialize_settings'
    /opt/gitlab/embedded/service/gitlab-rails/config/initializers/asset_proxy_settings.rb:5:in `block in <top (required)>'
    /opt/gitlab/embedded/service/gitlab-rails/config/initializers/asset_proxy_settings.rb:4:in `<top (required)>'
    /opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:6:in `<top (required)>'
    /opt/gitlab/embedded/bin/bundle:23:in `load'
    /opt/gitlab/embedded/bin/bundle:23:in `<main>'
    Tasks: TOP => gitlab:db:configure => environment
    (See full trace by running task with --trace)
    STDERR: 
    ---- End output of "bash"  "/tmp/chef-script20190919-19942-dd06t1" ----
    Ran "bash"  "/tmp/chef-script20190919-19942-dd06t1" returned 1
    
    Resource Declaration:
    ---------------------
    # In /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb
    
     54: bash "migrate gitlab-rails database" do
     55:   code <<-EOH
     56:     set -e
     57:     log_file="#{node['gitlab']['gitlab-rails']['log_directory']}/gitlab-rails-db-migrate-$(date +%Y-%m-%d-%H-%M-%S).log"
     58:     umask 077
     59:     /opt/gitlab/bin/gitlab-rake gitlab:db:configure 2>& 1 | tee ${log_file}
     60:     STATUS=${PIPESTATUS[0]}
     61:     chown #{account_helper.gitlab_user}:#{account_helper.gitlab_group} ${log_file}
     62:     echo $STATUS > #{db_migrate_status_file}
     63:     exit $STATUS
     64:   EOH
     65:   environment env_variables unless env_variables.empty?
     66:   notifies :run, "execute[clear the gitlab-rails cache]", :immediately
     67:   dependent_services.each do |svc|
     68:     notifies :restart, svc, :immediately
     69:   end
     70:   not_if "(test -f #{db_migrate_status_file}) && (cat #{db_migrate_status_file} | grep -Fx 0)"
     71:   only_if { node['gitlab']['gitlab-rails']['auto_migrate'] }
     72: end
     73: 
    
    Compiled Resource:
    ------------------
    # Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb:54:in `from_file'
    
    bash("migrate gitlab-rails database") do
      action [:run]
      default_guard_interpreter :default
      command nil
      backup 5
      interpreter "bash"
      declared_type :bash
      cookbook_name "gitlab"
      recipe_name "database_migrations"
      code "    set -e\n    log_file=\"/var/log/gitlab/gitlab-rails/gitlab-rails-db-migrate-$(date +%Y-%m-%d-%H-%M-%S).log\"\n    umask 077\n    /opt/gitlab/bin/gitlab-rake gitlab:db:configure 2>& 1 | tee ${log_file}\n    STATUS=${PIPESTATUS[0]}\n    chown git:git ${log_file}\n    echo $STATUS > /var/opt/gitlab/gitlab-rails/upgrade-status/db-migrate-873248b1f0d3a7a5535771a3a1635803-09f8edbc29a\n    exit $STATUS\n"
      domain nil
      user nil
      not_if "(test -f /var/opt/gitlab/gitlab-rails/upgrade-status/db-migrate-873248b1f0d3a7a5535771a3a1635803-09f8edbc29a) && (cat /var/opt/gitlab/gitlab-rails/upgrade-status/db-migrate-873248b1f0d3a7a5535771a3a1635803-09f8edbc29a | grep -Fx 0)"
      only_if { #code block }
    end
    
    System Info:
    ------------
    chef_version=14.13.11
    platform=centos
    platform_version=7.7.1908
    ruby=ruby 2.6.3p62 (2019-04-16 revision 67580) [x86_64-linux]
    program_name=/opt/gitlab/embedded/bin/chef-client
    executable=/opt/gitlab/embedded/bin/chef-client
    

Running handlers:
There was an error running gitlab-ctl reconfigure:

bash[migrate gitlab-rails database] (gitlab::database_migrations line 54) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of "bash"  "/tmp/chef-script20190919-19942-dd06t1" ----
STDOUT: rake aborted!
ArgumentError: wrong number of arguments (given 2, expected 0)
(eval):1:in `store'
/opt/gitlab/embedded/service/gitlab-rails/lib/banzai/filter/asset_proxy_filter.rb:41:in `initialize_settings'
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/asset_proxy_settings.rb:5:in `block in <top (required)>'
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/asset_proxy_settings.rb:4:in `<top (required)>'
/opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:6:in `<top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => gitlab:db:configure => environment
(See full trace by running task with --trace)
STDERR: 
---- End output of "bash"  "/tmp/chef-script20190919-19942-dd06t1" ----
Ran "bash"  "/tmp/chef-script20190919-19942-dd06t1" returned 1

Running handlers complete
Chef Client failed. 9 resources updated in 01 minutes 34 seconds

Can you run gitlab-ctl stop to stop and GitLab components currently running and attempt to reinstall GitLab using yum reinstall gitlab-ee?

I hope this is helpful. Let us know how it goes!

That was one of the first things I tried. The pre-upgrade checks fail because of this rake failure, so I reinstalled with rpm -Uvh --noscripts --force. No luck.

I have the problem narrowed down. I disabled my ldap block in /var/opt/gitlab/gitlab-rails/etc/gitlab.yml, and gitlab-rake started working again.

Obviously I still need to figure out how to re-enable LDAP without breaking everything, but this is progress…

What I was missing was that store in the errors above was my LDAP provider ID. Simply changing that to main and running gitlab-rake 'gitlab:ldap:rename_provider[ldapstore,ldapmain]' fixed the problem.

Whatever caused this to break seems to have happened between gitlab-ce-12.1.6-ce.0.el7.x86_64 and gitlab-ce-12.2.4-ce.0.el7.x86_64.