Hi,
Problem to solve
I’m running Gitlab CE in Docker on version 16.11.1-ce.0. When I tried to upgrade to 16.11.3-ce.0 I’ve got an error on the gitlab-ctl reconfigure for gitlab-rake cache:clear.
I tried to upgrade to 16.11.2-ce.0 first, but same error.
Here the latest log file error:
* execute[clear the gitlab-rails cache] action run…
[execute] rake aborted!…
ArgumentError: invalid log level: 2…
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/json_logger.rb:33:in `new'…
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/json_logger.rb:33:in `build'…
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/multi_destination_logger.rb:10:in `block in debug'…
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/multi_destination_logger.rb:10:in `each'…
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/multi_destination_logger.rb:10:in `debug'…
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/active_record_lifecycle.rb:12:in `block (2 levels) in <top (required)>'…
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/active_record_lifecycle.rb:7:in `block in <top (required)>'…
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/cluster/lifecycle_events.rb:91:in `on_worker_start'…
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/active_record_lifecycle.rb:6:in `<top (required)>'…
/opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:7:in `<top (required)>'…
<internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:37:in `require'…
<internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:37:in `require'…
/opt/gitlab/embedded/bin/bundle:25:in `load'…
/opt/gitlab/embedded/bin/bundle:25:in `<main>'…
Tasks: TOP => cache:clear => cache:clear:redis => environment…
(See full trace by running task with --trace)…
…
================================================================================…
Error executing action `run` on resource 'execute[clear the gitlab-rails cache]'…
================================================================================…
…
Mixlib::ShellOut::ShellCommandFailed…
------------------------------------…
Expected process to exit with [0], but received '1'…
---- Begin output of /opt/gitlab/bin/gitlab-rake cache:clear ----…
STDOUT: …
STDERR: rake aborted!…
ArgumentError: invalid log level: 2…
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/json_logger.rb:33:in `new'…
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/json_logger.rb:33:in `build'…
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/multi_destination_logger.rb:10:in `block in debug'…
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/multi_destination_logger.rb:10:in `each'…
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/multi_destination_logger.rb:10:in `debug'…
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/active_record_lifecycle.rb:12:in `block (2 levels) in <top (required)>'…
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/active_record_lifecycle.rb:7:in `block in <top (required)>'…
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/cluster/lifecycle_events.rb:91:in `on_worker_start'…
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/active_record_lifecycle.rb:6:in `<top (required)>'…
/opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:7:in `<top (required)>'…
<internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:37:in `require'…
<internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:37:in `require'…
/opt/gitlab/embedded/bin/bundle:25:in `load'…
/opt/gitlab/embedded/bin/bundle:25:in `<main>'…
Tasks: TOP => cache:clear => cache:clear:redis => environment…
(See full trace by running task with --trace)…
---- End output of /opt/gitlab/bin/gitlab-rake cache:clear ----…
Ran /opt/gitlab/bin/gitlab-rake cache:clear returned 1…
…
Resource Declaration:…
---------------------…
# In /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/gitlab-rails.rb…
…
543: execute "clear the gitlab-rails cache" do…
544: command "/opt/gitlab/bin/gitlab-rake cache:clear"…
545: action :nothing…
546: not_if { omnibus_helper.not_listening?('redis') || !node['gitlab']['gitlab_rails']['rake_cache_clear'] }…
547: end…
548: …
…
Compiled Resource:…
------------------…
# Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/gitlab-rails.rb:543:in `from_file'…
…
execute("clear the gitlab-rails cache") do…
action [:nothing]…
default_guard_interpreter :execute…
command "/opt/gitlab/bin/gitlab-rake cache:clear"…
declared_type :execute…
cookbook_name "gitlab"…
recipe_name "gitlab-rails"…
not_if { #code block }…
end…
…
System Info:…
------------…
chef_version=18.3.0…
platform=ubuntu…
platform_version=22.04…
ruby=ruby 3.1.4p223 (2023-03-30 revision 957bb7cb81) [x86_64-linux]…
program_name=/opt/gitlab/embedded/bin/cinc-client…
executable=/opt/gitlab/embedded/bin/cinc-client…
…
…
Running handlers:…
[2024-05-29T09:56:19+02:00] ERROR: Running exception handlers…
There was an error running gitlab-ctl reconfigure:…
…
Multiple failures occurred:…
* Mixlib::ShellOut::ShellCommandFailed occurred in Cinc Client run: rails_migration[gitlab-rails] (gitlab::database_migrations line 51) had an error: Mixlib::ShellOut::ShellC…
---- Begin output of "bash" ----…
STDOUT: rake aborted!…
ArgumentError: invalid log level: 2…
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/json_logger.rb:33:in `new'…
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/json_logger.rb:33:in `build'…
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/multi_destination_logger.rb:10:in `block in debug'…
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/multi_destination_logger.rb:10:in `each'…
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/multi_destination_logger.rb:10:in `debug'…
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/active_record_lifecycle.rb:12:in `block (2 levels) in <top (required)>'…
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/active_record_lifecycle.rb:7:in `block in <top (required)>'…
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/cluster/lifecycle_events.rb:91:in `on_worker_start'…
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/active_record_lifecycle.rb:6:in `<top (required)>'…
/opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:7:in `<top (required)>'…
<internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:37:in `require'…
<internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:37:in `require'…
/opt/gitlab/embedded/bin/bundle:25:in `load'…
/opt/gitlab/embedded/bin/bundle:25:in `<main>'…
Tasks: TOP => gitlab:db:configure => environment…
(See full trace by running task with --trace)…
STDERR: …
---- End output of "bash" ----…
Ran "bash" returned 1…
* Mixlib::ShellOut::ShellCommandFailed occurred in delayed notification: execute[clear the gitlab-rails cache] (gitlab::gitlab-rails line 543) had an error: Mixlib::ShellOut:…
---- Begin output of /opt/gitlab/bin/gitlab-rake cache:clear ----…
STDOUT: …
STDERR: rake aborted!…
ArgumentError: invalid log level: 2…
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/json_logger.rb:33:in `new'…
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/json_logger.rb:33:in `build'…
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/multi_destination_logger.rb:10:in `block in debug'…
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/multi_destination_logger.rb:10:in `each'…
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/multi_destination_logger.rb:10:in `debug'…
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/active_record_lifecycle.rb:12:in `block (2 levels) in <top (required)>'…
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/active_record_lifecycle.rb:7:in `block in <top (required)>'…
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/cluster/lifecycle_events.rb:91:in `on_worker_start'…
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/active_record_lifecycle.rb:6:in `<top (required)>'…
/opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:7:in `<top (required)>'…
<internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:37:in `require'…
<internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:37:in `require'…
/opt/gitlab/embedded/bin/bundle:25:in `load'…
/opt/gitlab/embedded/bin/bundle:25:in `<main>'…
Tasks: TOP => cache:clear => cache:clear:redis => environment…
(See full trace by running task with --trace)…
---- End output of /opt/gitlab/bin/gitlab-rake cache:clear ----…
Ran /opt/gitlab/bin/gitlab-rake cache:clear returned 1…
…
Running handlers complete…
[2024-05-29T09:56:19+02:00] ERROR: Exception handlers complete…
Infra Phase failed. 157 resources updated in 01 minutes 43 seconds…
[2024-05-29T09:56:19+02:00] FATAL: Stacktrace dumped to /opt/gitlab/embedded/cookbooks/cache/cinc-stacktrace.out…
[2024-05-29T09:56:19+02:00] FATAL: ---------------------------------------------------------------------------------------…
[2024-05-29T09:56:19+02:00] FATAL: PLEASE PROVIDE THE CONTENTS OF THE stacktrace.out FILE (above) IF YOU FILE A BUG REPORT…
[2024-05-29T09:56:19+02:00] FATAL: ---------------------------------------------------------------------------------------…
[2024-05-29T09:56:19+02:00] FATAL: Chef::Exceptions::MultipleFailures: Multiple failures occurred:…
* Mixlib::ShellOut::ShellCommandFailed occurred in Cinc Client run: rails_migration[gitlab-rails] (gitlab::database_migrations line 51) had an error: Mixlib::ShellOut::ShellC…
---- Begin output of "bash" ----…
STDOUT: rake aborted!…
ArgumentError: invalid log level: 2…
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/json_logger.rb:33:in `new'…
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/json_logger.rb:33:in `build'…
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/multi_destination_logger.rb:10:in `block in debug'…
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/multi_destination_logger.rb:10:in `each'…
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/multi_destination_logger.rb:10:in `debug'…
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/active_record_lifecycle.rb:12:in `block (2 levels) in <top (required)>'…
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/active_record_lifecycle.rb:7:in `block in <top (required)>'…
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/cluster/lifecycle_events.rb:91:in `on_worker_start'…
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/active_record_lifecycle.rb:6:in `<top (required)>'…
/opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:7:in `<top (required)>'…
<internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:37:in `require'…
<internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:37:in `require'…
/opt/gitlab/embedded/bin/bundle:25:in `load'…
/opt/gitlab/embedded/bin/bundle:25:in `<main>'…
Tasks: TOP => gitlab:db:configure => environment…
(See full trace by running task with --trace)…
STDERR: …
---- End output of "bash" ----…
Ran "bash" returned 1…
* Mixlib::ShellOut::ShellCommandFailed occurred in delayed notification: execute[clear the gitlab-rails cache] (gitlab::gitlab-rails line 543) had an error: Mixlib::ShellOut:…
---- Begin output of /opt/gitlab/bin/gitlab-rake cache:clear ----…
STDOUT: …
STDERR: rake aborted!…
ArgumentError: invalid log level: 2…
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/json_logger.rb:33:in `new'…
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/json_logger.rb:33:in `build'…
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/multi_destination_logger.rb:10:in `block in debug'…
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/multi_destination_logger.rb:10:in `each'…
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/multi_destination_logger.rb:10:in `debug'…
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/active_record_lifecycle.rb:12:in `block (2 levels) in <top (required)>'…
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/active_record_lifecycle.rb:7:in `block in <top (required)>'…
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/cluster/lifecycle_events.rb:91:in `on_worker_start'…
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/active_record_lifecycle.rb:6:in `<top (required)>'…
/opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:7:in `<top (required)>'…
<internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:37:in `require'…
<internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:37:in `require'…
/opt/gitlab/embedded/bin/bundle:25:in `load'…
/opt/gitlab/embedded/bin/bundle:25:in `<main>'…
Tasks: TOP => cache:clear => cache:clear:redis => environment…
(See full trace by running task with --trace)…
---- End output of /opt/gitlab/bin/gitlab-rake cache:clear ----…
Ran /opt/gitlab/bin/gitlab-rake cache:clear returned 1…
Steps to reproduce
I just replaced the image in my compose file from 16.11.1-ce.0 to 16.11.2-ce.0
services:
## GitLab
### Version control system
gitlab:
image: gitlab/gitlab-ce:16.11.2-ce.0
restart: always
container_name: gitlab
hostname: git.mydomain.local
environment:
- TZ=Europe/Paris
- GITLAB_LOG_LEVEL=2
ports:
- 2225:22
volumes:
- /mnt/docker/gitlab-stg/etc/gitlab:/etc/gitlab
- /mnt/docker/gitlab-stg/var/log/gitlab:/var/log/gitlab
- /mnt/docker/gitlab-stg/var/opt/gitlab:/var/opt/gitlab
shm_size: 256m
logging:
driver: json-file
networks:
gitlab:
ipv4_address: 192.168.5.30
networks:
gitlab:
external: true
Configuration
N/A
Versions
Please select whether options apply, and add the version information.
- Self-managed
- GitLab.com SaaS