Can't upgrade from 16.11.1-ce.0 in Docker

Hi,

Problem to solve

I’m running Gitlab CE in Docker on version 16.11.1-ce.0. When I tried to upgrade to 16.11.3-ce.0 I’ve got an error on the gitlab-ctl reconfigure for gitlab-rake cache:clear.

I tried to upgrade to 16.11.2-ce.0 first, but same error.

Here the latest log file error:

* execute[clear the gitlab-rails cache] action run…
    [execute] rake aborted!…
              ArgumentError: invalid log level: 2…
              /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/json_logger.rb:33:in `new'…
              /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/json_logger.rb:33:in `build'…
              /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/multi_destination_logger.rb:10:in `block in debug'…
              /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/multi_destination_logger.rb:10:in `each'…
              /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/multi_destination_logger.rb:10:in `debug'…
              /opt/gitlab/embedded/service/gitlab-rails/config/initializers/active_record_lifecycle.rb:12:in `block (2 levels) in <top (required)>'…
              /opt/gitlab/embedded/service/gitlab-rails/config/initializers/active_record_lifecycle.rb:7:in `block in <top (required)>'…
              /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/cluster/lifecycle_events.rb:91:in `on_worker_start'…
              /opt/gitlab/embedded/service/gitlab-rails/config/initializers/active_record_lifecycle.rb:6:in `<top (required)>'…
              /opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:7:in `<top (required)>'…
              <internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:37:in `require'…
              <internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:37:in `require'…
              /opt/gitlab/embedded/bin/bundle:25:in `load'…
              /opt/gitlab/embedded/bin/bundle:25:in `<main>'…
              Tasks: TOP => cache:clear => cache:clear:redis => environment…
              (See full trace by running task with --trace)…
    …
    ================================================================================…
    Error executing action `run` on resource 'execute[clear the gitlab-rails cache]'…
    ================================================================================…
    …
    Mixlib::ShellOut::ShellCommandFailed…
    ------------------------------------…
    Expected process to exit with [0], but received '1'…
    ---- Begin output of /opt/gitlab/bin/gitlab-rake cache:clear ----…
    STDOUT: …
    STDERR: rake aborted!…
    ArgumentError: invalid log level: 2…
    /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/json_logger.rb:33:in `new'…
    /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/json_logger.rb:33:in `build'…
    /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/multi_destination_logger.rb:10:in `block in debug'…
    /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/multi_destination_logger.rb:10:in `each'…
    /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/multi_destination_logger.rb:10:in `debug'…
    /opt/gitlab/embedded/service/gitlab-rails/config/initializers/active_record_lifecycle.rb:12:in `block (2 levels) in <top (required)>'…
    /opt/gitlab/embedded/service/gitlab-rails/config/initializers/active_record_lifecycle.rb:7:in `block in <top (required)>'…
    /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/cluster/lifecycle_events.rb:91:in `on_worker_start'…
    /opt/gitlab/embedded/service/gitlab-rails/config/initializers/active_record_lifecycle.rb:6:in `<top (required)>'…
    /opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:7:in `<top (required)>'…
    <internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:37:in `require'…
    <internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:37:in `require'…
    /opt/gitlab/embedded/bin/bundle:25:in `load'…
    /opt/gitlab/embedded/bin/bundle:25:in `<main>'…
    Tasks: TOP => cache:clear => cache:clear:redis => environment…
    (See full trace by running task with --trace)…
    ---- End output of /opt/gitlab/bin/gitlab-rake cache:clear ----…
    Ran /opt/gitlab/bin/gitlab-rake cache:clear returned 1…
    …
    Resource Declaration:…
    ---------------------…
    # In /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/gitlab-rails.rb…
    …
    543: execute "clear the gitlab-rails cache" do…
    544:   command "/opt/gitlab/bin/gitlab-rake cache:clear"…
    545:   action :nothing…
    546:   not_if { omnibus_helper.not_listening?('redis') || !node['gitlab']['gitlab_rails']['rake_cache_clear'] }…
    547: end…
    548: …
    …
    Compiled Resource:…
    ------------------…
    # Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/gitlab-rails.rb:543:in `from_file'…
    …
    execute("clear the gitlab-rails cache") do…
      action [:nothing]…
      default_guard_interpreter :execute…
      command "/opt/gitlab/bin/gitlab-rake cache:clear"…
      declared_type :execute…
      cookbook_name "gitlab"…
      recipe_name "gitlab-rails"…
      not_if { #code block }…
    end…
    …
    System Info:…
    ------------…
    chef_version=18.3.0…
    platform=ubuntu…
    platform_version=22.04…
    ruby=ruby 3.1.4p223 (2023-03-30 revision 957bb7cb81) [x86_64-linux]…
    program_name=/opt/gitlab/embedded/bin/cinc-client…
    executable=/opt/gitlab/embedded/bin/cinc-client…
    …
…
Running handlers:…
[2024-05-29T09:56:19+02:00] ERROR: Running exception handlers…
There was an error running gitlab-ctl reconfigure:…
…
Multiple failures occurred:…
* Mixlib::ShellOut::ShellCommandFailed occurred in Cinc Client run: rails_migration[gitlab-rails] (gitlab::database_migrations line 51) had an error: Mixlib::ShellOut::ShellC…
---- Begin output of "bash"  ----…
STDOUT: rake aborted!…
ArgumentError: invalid log level: 2…
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/json_logger.rb:33:in `new'…
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/json_logger.rb:33:in `build'…
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/multi_destination_logger.rb:10:in `block in debug'…
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/multi_destination_logger.rb:10:in `each'…
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/multi_destination_logger.rb:10:in `debug'…
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/active_record_lifecycle.rb:12:in `block (2 levels) in <top (required)>'…
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/active_record_lifecycle.rb:7:in `block in <top (required)>'…
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/cluster/lifecycle_events.rb:91:in `on_worker_start'…
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/active_record_lifecycle.rb:6:in `<top (required)>'…
/opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:7:in `<top (required)>'…
<internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:37:in `require'…
<internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:37:in `require'…
/opt/gitlab/embedded/bin/bundle:25:in `load'…
/opt/gitlab/embedded/bin/bundle:25:in `<main>'…
Tasks: TOP => gitlab:db:configure => environment…
(See full trace by running task with --trace)…
STDERR: …
---- End output of "bash"  ----…
Ran "bash"  returned 1…
* Mixlib::ShellOut::ShellCommandFailed occurred in delayed notification: execute[clear the gitlab-rails cache] (gitlab::gitlab-rails line 543) had an error: Mixlib::ShellOut:…
---- Begin output of /opt/gitlab/bin/gitlab-rake cache:clear ----…
STDOUT: …
STDERR: rake aborted!…
ArgumentError: invalid log level: 2…
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/json_logger.rb:33:in `new'…
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/json_logger.rb:33:in `build'…
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/multi_destination_logger.rb:10:in `block in debug'…
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/multi_destination_logger.rb:10:in `each'…
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/multi_destination_logger.rb:10:in `debug'…
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/active_record_lifecycle.rb:12:in `block (2 levels) in <top (required)>'…
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/active_record_lifecycle.rb:7:in `block in <top (required)>'…
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/cluster/lifecycle_events.rb:91:in `on_worker_start'…
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/active_record_lifecycle.rb:6:in `<top (required)>'…
/opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:7:in `<top (required)>'…
<internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:37:in `require'…
<internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:37:in `require'…
/opt/gitlab/embedded/bin/bundle:25:in `load'…
/opt/gitlab/embedded/bin/bundle:25:in `<main>'…
Tasks: TOP => cache:clear => cache:clear:redis => environment…
(See full trace by running task with --trace)…
---- End output of /opt/gitlab/bin/gitlab-rake cache:clear ----…
Ran /opt/gitlab/bin/gitlab-rake cache:clear returned 1…
…
Running handlers complete…
[2024-05-29T09:56:19+02:00] ERROR: Exception handlers complete…
Infra Phase failed. 157 resources updated in 01 minutes 43 seconds…
[2024-05-29T09:56:19+02:00] FATAL: Stacktrace dumped to /opt/gitlab/embedded/cookbooks/cache/cinc-stacktrace.out…
[2024-05-29T09:56:19+02:00] FATAL: ---------------------------------------------------------------------------------------…
[2024-05-29T09:56:19+02:00] FATAL: PLEASE PROVIDE THE CONTENTS OF THE stacktrace.out FILE (above) IF YOU FILE A BUG REPORT…
[2024-05-29T09:56:19+02:00] FATAL: ---------------------------------------------------------------------------------------…
[2024-05-29T09:56:19+02:00] FATAL: Chef::Exceptions::MultipleFailures: Multiple failures occurred:…
* Mixlib::ShellOut::ShellCommandFailed occurred in Cinc Client run: rails_migration[gitlab-rails] (gitlab::database_migrations line 51) had an error: Mixlib::ShellOut::ShellC…
---- Begin output of "bash"  ----…
STDOUT: rake aborted!…
ArgumentError: invalid log level: 2…
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/json_logger.rb:33:in `new'…
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/json_logger.rb:33:in `build'…
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/multi_destination_logger.rb:10:in `block in debug'…
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/multi_destination_logger.rb:10:in `each'…
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/multi_destination_logger.rb:10:in `debug'…
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/active_record_lifecycle.rb:12:in `block (2 levels) in <top (required)>'…
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/active_record_lifecycle.rb:7:in `block in <top (required)>'…
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/cluster/lifecycle_events.rb:91:in `on_worker_start'…
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/active_record_lifecycle.rb:6:in `<top (required)>'…
/opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:7:in `<top (required)>'…
<internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:37:in `require'…
<internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:37:in `require'…
/opt/gitlab/embedded/bin/bundle:25:in `load'…
/opt/gitlab/embedded/bin/bundle:25:in `<main>'…
Tasks: TOP => gitlab:db:configure => environment…
(See full trace by running task with --trace)…
STDERR: …
---- End output of "bash"  ----…
Ran "bash"  returned 1…
* Mixlib::ShellOut::ShellCommandFailed occurred in delayed notification: execute[clear the gitlab-rails cache] (gitlab::gitlab-rails line 543) had an error: Mixlib::ShellOut:…
---- Begin output of /opt/gitlab/bin/gitlab-rake cache:clear ----…
STDOUT: …
STDERR: rake aborted!…
ArgumentError: invalid log level: 2…
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/json_logger.rb:33:in `new'…
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/json_logger.rb:33:in `build'…
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/multi_destination_logger.rb:10:in `block in debug'…
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/multi_destination_logger.rb:10:in `each'…
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/multi_destination_logger.rb:10:in `debug'…
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/active_record_lifecycle.rb:12:in `block (2 levels) in <top (required)>'…
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/active_record_lifecycle.rb:7:in `block in <top (required)>'…
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/cluster/lifecycle_events.rb:91:in `on_worker_start'…
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/active_record_lifecycle.rb:6:in `<top (required)>'…
/opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:7:in `<top (required)>'…
<internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:37:in `require'…
<internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:37:in `require'…
/opt/gitlab/embedded/bin/bundle:25:in `load'…
/opt/gitlab/embedded/bin/bundle:25:in `<main>'…
Tasks: TOP => cache:clear => cache:clear:redis => environment…
(See full trace by running task with --trace)…
---- End output of /opt/gitlab/bin/gitlab-rake cache:clear ----…
Ran /opt/gitlab/bin/gitlab-rake cache:clear returned 1…

Steps to reproduce

I just replaced the image in my compose file from 16.11.1-ce.0 to 16.11.2-ce.0

services:
  ## GitLab
  ### Version control system
  gitlab:
    image: gitlab/gitlab-ce:16.11.2-ce.0
    restart: always
    container_name: gitlab
    hostname: git.mydomain.local
    environment:
      - TZ=Europe/Paris
      - GITLAB_LOG_LEVEL=2
    ports:
      - 2225:22
    volumes:
      - /mnt/docker/gitlab-stg/etc/gitlab:/etc/gitlab
      - /mnt/docker/gitlab-stg/var/log/gitlab:/var/log/gitlab
      - /mnt/docker/gitlab-stg/var/opt/gitlab:/var/opt/gitlab
    shm_size: 256m
    logging:
      driver: json-file
    networks:
      gitlab:
        ipv4_address: 192.168.5.30
networks:
  gitlab:
    external: true

Configuration

N/A

Versions

Please select whether options apply, and add the version information.