Updating from 15.11 to latest docker build fails

Hi!

I have been updating gitlab monthly on the 23rd using the official docker build gitlab/gitlab-ce for a couple of years without issue. This time, it fails on the command docker-compose exec gitlab gitlab-ctl reconfigure

Any suggestions welcome! I also tried updating to the latest 15.11.5ce version first; that worked but the upgrade to 16 failed.

[2023-05-23T21:43:08+00:00] INFO: Started Cinc Zero at chefzero://localhost:1 with repository at /opt/gitlab/embedded (One version per cookbook)
Cinc Client, version 17.10.0
Patents: https://www.chef.io/patents
Infra Phase starting
[2023-05-23T21:43:08+00:00] INFO: *** Cinc Client 17.10.0 ***
[2023-05-23T21:43:08+00:00] INFO: Platform: x86_64-linux
[2023-05-23T21:43:08+00:00] INFO: Cinc-client pid: 21
[2023-05-23T21:43:09+00:00] INFO: Setting the run_list to ["recipe[gitlab]"] from CLI options
[2023-05-23T21:43:09+00:00] INFO: Run List is [recipe[gitlab]]
[2023-05-23T21:43:09+00:00] INFO: Run List expands to [gitlab]
[2023-05-23T21:43:09+00:00] INFO: Starting Cinc Client Run for repo.integralagile.com
[2023-05-23T21:43:09+00:00] INFO: Running start handlers
[2023-05-23T21:43:09+00:00] INFO: Start handlers complete.
Resolving cookbooks for run list: ["gitlab"]
[2023-05-23T21:43:10+00:00] INFO: Loading cookbooks [gitlab@0.0.1, package@0.1.0, logrotate@0.1.0, postgresql@0.1.0, redis@0.1.0, monitoring@0.1.0, registry@0.1.0, mattermost@0.1.0, consul@0.1.0, gitaly@0.1.0, praefect@0.1.0, gitlab-kas@0.1.0, gitlab-pages@0.1.0, letsencrypt@0.1.0, nginx@0.1.0, runit@5.1.7, acme@4.1.6, crond@0.1.0]
Synchronizing cookbooks:

  ================================================================================
  Error Syncing Cookbooks:
  ================================================================================

  Unexpected Error:
  -----------------
  ThreadError: can't create Thread: Operation not permitted

  System Info:
  ------------
  chef_version=17.10.0
  platform=ubuntu
  platform_version=22.04
  ruby=ruby 3.0.6p216 (2023-03-30 revision 23a532679b) [x86_64-linux]
  program_name=/opt/gitlab/embedded/bin/cinc-client
  executable=/opt/gitlab/embedded/bin/cinc-client


  Running handlers:
[2023-05-23T21:43:10+00:00] ERROR: Running exception handlers
There was an error running gitlab-ctl reconfigure:

can't create Thread: Operation not permitted
[2023-05-23T20:38:55+00:00] ERROR: Report handler GitLabHandler::Exception raised #<NameError: uninitialized constant GitLabHandler::Exception::OmnibusHelper>
[2023-05-23T20:38:55+00:00] ERROR: /opt/gitlab/embedded/cookbooks/package/libraries/handlers/gitlab.rb:30:in `report'
[2023-05-23T20:38:55+00:00] ERROR: /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/handler.rb:296:in `run_report_unsafe'
[2023-05-23T20:38:55+00:00] ERROR: /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/handler.rb:284:in `run_report_safely'
[2023-05-23T20:38:55+00:00] ERROR: /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/handler.rb:155:in `block in run_exception_handlers'
[2023-05-23T20:38:55+00:00] ERROR: /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/handler.rb:153:in `each'
[2023-05-23T20:38:55+00:00] ERROR: /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/handler.rb:153:in `run_exception_handlers'
[2023-05-23T20:38:55+00:00] ERROR: /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/handler.rb:164:in `block in <class:Handler>'
[2023-05-23T20:38:55+00:00] ERROR: /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/client.rb:443:in `block in run_failed'
[2023-05-23T20:38:55+00:00] ERROR: /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/client.rb:442:in `each'
[2023-05-23T20:38:55+00:00] ERROR: /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/client.rb:442:in `run_failed'
[2023-05-23T20:38:55+00:00] ERROR: /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/client.rb:306:in `rescue in run'
[2023-05-23T20:38:55+00:00] ERROR: /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/client.rb:299:in `run'
[2023-05-23T20:38:55+00:00] ERROR: /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/application.rb:305:in `run_with_graceful_exit_option'
[2023-05-23T20:38:55+00:00] ERROR: /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/application.rb:281:in `block in run_chef_client'
[2023-05-23T20:38:55+00:00] ERROR: /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/local_mode.rb:42:in `with_server_connectivity'
[2023-05-23T20:38:55+00:00] ERROR: /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/application.rb:264:in `run_chef_client'
[2023-05-23T20:38:55+00:00] ERROR: /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/application/base.rb:352:in `run_application'
[2023-05-23T20:38:55+00:00] ERROR: /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/application.rb:67:in `run'
[2023-05-23T20:38:55+00:00] ERROR: /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-bin-17.10.0/bin/cinc-client:25:in `<top (required)>'
[2023-05-23T20:38:55+00:00] ERROR: /opt/gitlab/embedded/bin/cinc-client:25:in `load'
[2023-05-23T20:38:55+00:00] ERROR: /opt/gitlab/embedded/bin/cinc-client:25:in `<main>'
  Running handlers complete
[2023-05-23T20:38:55+00:00] ERROR: Exception handlers complete
  Infra Phase failed. 0 resources updated in 01 seconds
[2023-05-23T20:38:55+00:00] FATAL: Stacktrace dumped to /opt/gitlab/embedded/cookbooks/cache/cinc-stacktrace.out
[2023-05-23T20:38:55+00:00] FATAL: ---------------------------------------------------------------------------------------
[2023-05-23T20:38:55+00:00] FATAL: PLEASE PROVIDE THE CONTENTS OF THE stacktrace.out FILE (above) IF YOU FILE A BUG REPORT
[2023-05-23T20:38:55+00:00] FATAL: ---------------------------------------------------------------------------------------
[2023-05-23T20:38:55+00:00] FATAL: ThreadError: can't create Thread: Operation not permitted
UPDATE COMPLETE - use docker logs -f gitlab for SHOWING STARTUP LOGS

same here,
i also have an old, historically grown installation.
Last update was from 15.11.0 to 15.11.5 (which still works)
but update to v16.0.1 fails with the exact same error message

same here. Coming from the 15.11.2-ce.0

same story here

It seems to depend on the ‘history’ or maybe number of projects?
On another, smaller gitlab the update to v16 was without any problems.

The v16 errors indicate potential problems with access rights, but even if you run the recommended ‘update-permissions’ or reconfigure gitlab… this all does not help in starting v16

Did anyone open a bugticket at gitlab yet?

I tried, but getting a page not found when I click that link in the support page: https://gitlab.com/gitlab-org/gitlab/issues.

Does anyone know the correct link?

link seems ok to me?

I opened a ticket

Weird, ok thanks! I also thought about setting up a clean gitlab-16 docker then migrating … but migration requires both running the same version (doh!).

Summary from the ticket, so we have it here as well:

It appears that the new GitLab 16.0 (which updated to use ubuntu 22.04 as the base image), now requires to be run in docker 20.10.10 or newer

For many reasons, we cannot upgrade the docker engine (currently the 19.3).

Due to this issue we cannot move to Gitlab 16.

I have the same problem - Unsupported Netgear ReadyNAS running Debian Jessie and cannot upgrade docker engine past 19.3.

Not surprising, Debian Jessie (Debian 8), is really old and EOL since June 2020. If you want to use newer versions of Gitlab, then you need to upgrade the host system that is underneath it. If you cannot do that on the Netgear ReadyNAS then you need to look at buying a new system, or a cheaper way create a VPS on a hosting service.