Upgrading GitLab CE Docker Image from 16.5.3-ce.0 to 16.6.1-ce.0 Fails

Hello GitLab Community,

I am facing a challenge while trying to upgrade my GitLab CE Docker image from version 16.5.3-ce.0 to 16.6.1-ce.0.

My current working instance is configured as follows:

docker-compose.yml

version: '2'
services:
  gitlab:
    image: gitlab/gitlab-ce:16.5.3-ce.0
    environment:
      GITLAB_OMNIBUS_CONFIG: |-
        external_url '[--redacted-for-privacy--]';
        gitlab_rails['gitlab_ssh_host'] = '[--redacted-for-privacy--]';
        gitlab_rails['gitlab_shell_ssh_port'] = 8500;
        gitlab_rails['gitlab_default_theme'] = 11;
        gitlab_rails['lfs_enabled'] = true;
        gitlab_rails['monitoring_whitelist'] = ['[--redacted-for-privacy--]'];
        gitlab_rails['gitlab_email_from'] = 'notifications@[--redacted-for-privacy--]';
        gitlab_rails['gitlab_email_display_name'] = '[--redacted-for-privacy--]';
        gitlab_rails['incoming_email_enabled'] = false;
        gitlab_rails['smtp_enable'] = true;
        gitlab_rails['smtp_address'] = "[--redacted-for-privacy--]";
        gitlab_rails['smtp_port'] = [--redacted-for-privacy--];
        gitlab_rails['smtp_user_name'] = "[--redacted-for-privacy--]";
        gitlab_rails['smtp_password'] = "[--redacted-for-privacy--]";
        gitlab_rails['smtp_domain'] = "[--redacted-for-privacy--]";
        gitlab_rails['smtp_authentication'] = "login";
        gitlab_rails['smtp_enable_starttls_auto'] = [--redacted-for-privacy--];
        gitlab_rails['smtp_tls'] = [--redacted-for-privacy--];
        gitlab_rails['smtp_openssl_verify_mode'] = 'peer';
        gitlab_rails['omniauth_allow_single_sign_on'] = false;
        gitlab_rails['omniauth_block_auto_created_users'] = true;
        gitlab_rails['omniauth_providers'] = [          {            "name" => "github",            "app_id" => "[--redacted-for-privacy--]",            "app_secret" => "[--redacted-for-privacy--]",            "args" => { gitlab_username_claim: 'name', "scope" => "user:email" }          },          {            "name": "google_oauth2",            "app_id": "[--redacted-for-privacy--]",            "app_secret": "[--redacted-for-privacy--]",            "args": { gitlab_username_claim: 'name', access_type: "offline", approval_prompt: "" }          }        ];
        gitlab_rails['object_store']['enabled'] = false;
        nginx['enable'] = true;
        nginx['listen_port'] = 80;
        nginx['listen_https'] = false;
        pages_nginx['enable'] = true;
        pages_external_url "[--redacted-for-privacy--]";
        gitlab_pages['access_control'] = true;
        pages_nginx['listen_https'] = false;
        pages_nginx['listen_port'] = 9000;
        pages_nginx['redirect_http_to_https'] = false;
        gitlab_pages['enable'] = true;
        gitlab_pages['redirect_http'] = false;
        gitlab_pages['inplace_chroot'] = true;
        gitlab_pages['external_http'] = [];
        gitlab_pages['listen_proxy'] = "0.0.0.0:8090";
        registry_external_url 'https://[--redacted-for-privacy--]/';
        registry_nginx['listen_port'] = 8000;
        registry_nginx['listen_https'] = false;
        registry_nginx['redirect_http_to_https'] = false;
        mattermost['enable'] = false;
        mattermost_nginx['enable'] = false;
        grafana['enable'] = false;
        letsencrypt['enable'] = false;
        prometheus['enable'] = false;
        alertmanager['enable'] = true;
        gitlab_exporter['enable'] = true;
        gitlab_pages['enable'] = true;
        gitlab_workhorse['enable'] = true;
        logrotate['enable'] = true;
        node_exporter['enable'] = true;
        postgres_exporter['enable'] = true;
        postgresql['enable'] = true;
        puma['enable'] = true;
        redis['enable'] = true;
        redis_exporter['enable'] = true;
        registry['enable'] = true;
        sidekiq['enable'] = true;
    stdin_open: true
    volumes:
    - /[--redacted-for-privacy--]/data:/var/opt/gitlab
    - /[--redacted-for-privacy--]/git-data/:/var/opt/gitlab/git-data/:rw
    - /[--redacted-for-privacy--]/shared/:/var/opt/gitlab/gitlab-rails/shared/:rw
    - /[--redacted-for-privacy--]/builds/:/var/opt/gitlab/gitlab-ci/builds:rw
    - /[--redacted-for-privacy--]/uploads/:/var/opt/gitlab/gitlab-rails/uploads:rw
    - /[--redacted-for-privacy--]/ssh/:/var/opt/gitlab/.ssh:rw
    - /[--redacted-for-privacy--]/conf-files:/etc/gitlab
    - /[--redacted-for-privacy--]/log-data:/var/log/gitlab
    logging:
      driver: json-file
      options:
        max-file: '2'
        max-size: 25m
    ports:
    - [--redacted-for-privacy--]:22/tcp

When I attempt to upgrade the Docker image and it starts, the only log output I get is:

6/12/2023 21:04:46Thank you for using GitLab Docker Image!
6/12/2023 21:04:46Current version: gitlab-ce=16.6.1-ce.0
6/12/2023 21:04:46
6/12/2023 21:04:46Configure GitLab for your system by editing /etc/gitlab/gitlab.rb file
6/12/2023 21:04:46And restart this container to reload settings.
6/12/2023 21:04:46To do it use docker exec:
6/12/2023 21:04:46
6/12/2023 21:04:46  docker exec -it gitlab editor /etc/gitlab/gitlab.rb
6/12/2023 21:04:46  docker restart gitlab
6/12/2023 21:04:46
6/12/2023 21:04:46For a comprehensive list of configuration options please see the Omnibus GitLab readme
6/12/2023 21:04:46https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/README.md
6/12/2023 21:04:46
6/12/2023 21:04:46If this container fails to start due to permission problems try to fix it by executing:
6/12/2023 21:04:46
6/12/2023 21:04:46  docker exec -it gitlab update-permissions
6/12/2023 21:04:46  docker restart gitlab
6/12/2023 21:04:46
6/12/2023 21:04:49Cleaning stale PIDs & sockets

This issue persists regardless of whether I use version 16.6.0 or 16.6.1.

My environment is Docker 19.03.15, running on Ubuntu 20.04.6 LTS (5.15.0). I’m using local volumes to persist repo data, DB, logs and LFS.
Over the past four years, my upgrade strategy has consistently been to move from the latest minor version to the earliest major version. This approach has served me well, and I’ve encountered no issues until now.

To illustrate, if I were on version 15.11.13-ce.0, my next upgrade would be to 16.0.0-ce.0, then to 16.0.8-ce.0, and so forth. I have followed this pattern successfully for all my upgrades.

I am at a loss as to what could be causing this issue. Could anyone please shed some light on what might be going wrong? Any help would be greatly appreciated.

Thank you all in advance.

Quick update.
I just went straight to the “wait for it” strategy. Tryed to upgrade to v16.6.2 and it worked as usual.