Upgrade failed from 18.1.2 to 18.2

How to Use GitLab Upgrade
1

I have an ubuntu server 22.04.5 LTS running self-hosted GitLab instance. Today I tried to update the instance to the latest version using apt upgrade. The update failed on cookcook compilation. Apparently it tries to install recipes from gitlab-ee even though i have gitlab-ce installed.

What steps could I do to resolve this issue?

Here is the full ouput from the console

Unpacking gitlab-ce (18.2.0-ce.0) over (18.1.2-ce.0) ...
Setting up gitlab-ce (18.2.0-ce.0) ...
Checking PostgreSQL executables:[2025-07-18T08:27:11+03:00] INFO: Started Cinc Zero at chefzero://localhost:1 with repository at /opt/gitlab/embedded (One version per cookbook)
Cinc Client, version 18.3.0
Patents: https://www.chef.io/patents
Infra Phase starting
[2025-07-18T08:27:11+03:00] INFO: *** Cinc Client 18.3.0 ***
[2025-07-18T08:27:11+03:00] INFO: Platform: x86_64-linux
[2025-07-18T08:27:11+03:00] INFO: Cinc-client pid: 937509
[2025-07-18T08:27:13+03:00] INFO: Setting the run_list to ["recipe[gitlab::config]", "recipe[postgresql::bin]"] from CLI options
[2025-07-18T08:27:13+03:00] INFO: Run List is [recipe[gitlab::config], recipe[postgresql::bin]]
[2025-07-18T08:27:13+03:00] INFO: Run List expands to [gitlab::config, postgresql::bin]
[2025-07-18T08:27:13+03:00] INFO: Starting Cinc Client Run for git.company.tld
[2025-07-18T08:27:13+03:00] INFO: Running start handlers
[2025-07-18T08:27:13+03:00] INFO: Start handlers complete.
Resolving cookbooks for run list: ["gitlab::config", "postgresql::bin"]
[2025-07-18T08:27:14+03:00] INFO: Loading cookbooks [gitlab@0.0.1, postgresql@0.1.0, package@0.1.0, logrotate@0.1.0, redis@0.1.0, monitoring@0.1.0, registry@0.1.0, mattermost@0.1.0, consul@0.1.0, gitaly@0.1.0, praefect@0.1.0, gitlab-kas@0.1.0, gitlab-pages@0.1.0, letsencrypt@0.1.0, nginx@0.1.0, runit@5.1.7, acme@4.1.6, crond@0.1.0]
[2025-07-18T08:27:14+03:00] INFO: Removing cookbooks/gitlab/libraries/account_helper.rb from the cache; its is no longer in the cookbook manifest.
[2025-07-18T08:27:14+03:00] INFO: Removing cookbooks/gitlab/libraries/helpers/base_helper.rb from the cache; its is no longer in the cookbook manifest.
[2025-07-18T08:27:14+03:00] INFO: Removing cookbooks/gitlab/libraries/logfiles_helper.rb from the cache; its is no longer in the cookbook manifest.
[2025-07-18T08:27:14+03:00] INFO: Removing cookbooks/gitlab/libraries/redis_uri.rb from the cache; its is no longer in the cookbook manifest.
Synchronizing cookbooks:
[2025-07-18T08:27:14+03:00] INFO: Storing updated cookbooks/gitlab/templates/default/gitconfig.erb in the cache.
[2025-07-18T08:27:14+03:00] INFO: Storing updated cookbooks/gitlab/templates/default/nginx-gitlab-http.conf.erb in the cache.
[2025-07-18T08:27:14+03:00] INFO: Storing updated cookbooks/gitlab/recipes/default.rb in the cache.
[2025-07-18T08:27:14+03:00] INFO: Storing updated cookbooks/gitlab/libraries/gitlab_exporter.rb in the cache.
[2025-07-18T08:27:14+03:00] INFO: Storing updated cookbooks/gitlab/libraries/gitlab_workhorse.rb in the cache.
[2025-07-18T08:27:14+03:00] INFO: Storing updated cookbooks/gitlab/recipes/users.rb in the cache.
[2025-07-18T08:27:14+03:00] INFO: Storing updated cookbooks/gitlab/libraries/helpers/gitlab_rails.rb in the cache.
[2025-07-18T08:27:14+03:00] INFO: Storing updated cookbooks/gitlab/libraries/helpers/base_pg_helper.rb in the cache.
[2025-07-18T08:27:14+03:00] INFO: Storing updated cookbooks/gitlab/libraries/redis.rb in the cache.
[2025-07-18T08:27:14+03:00] INFO: Storing updated cookbooks/gitlab/libraries/helpers/pg_status_helper.rb in the cache.
[2025-07-18T08:27:14+03:00] INFO: Storing updated cookbooks/gitlab/libraries/helpers/gitlab_workhorse_helper.rb in the cache.
[2025-07-18T08:27:14+03:00] INFO: Storing updated cookbooks/gitlab/attributes/default.rb in the cache.
  - gitlab (0.0.1)
  - postgresql (0.1.0)
[2025-07-18T08:27:14+03:00] INFO: Storing updated cookbooks/package/libraries/config/gitlab.rb in the cache.
[2025-07-18T08:27:14+03:00] INFO: Storing updated cookbooks/package/libraries/certificate_helper.rb in the cache.
[2025-07-18T08:27:14+03:00] INFO: Storing updated cookbooks/package/libraries/account_helper.rb in the cache.
[2025-07-18T08:27:14+03:00] INFO: Storing updated cookbooks/package/libraries/helpers/base_helper.rb in the cache.
[2025-07-18T08:27:14+03:00] INFO: Storing updated cookbooks/package/libraries/deprecations.rb in the cache.
[2025-07-18T08:27:14+03:00] INFO: Storing updated cookbooks/package/libraries/helpers/redis_helper/base.rb in the cache.
[2025-07-18T08:27:14+03:00] INFO: Storing updated cookbooks/package/libraries/helpers/redis_helper.rb in the cache.
[2025-07-18T08:27:14+03:00] INFO: Storing updated cookbooks/package/libraries/redis_uri.rb in the cache.
[2025-07-18T08:27:14+03:00] INFO: Storing updated cookbooks/package/libraries/logfiles_helper.rb in the cache.
[2025-07-18T08:27:14+03:00] INFO: Storing updated cookbooks/package/libraries/omnibus_helper.rb in the cache.
  - logrotate (0.1.0)
  - package (0.1.0)
  - redis (0.1.0)
[2025-07-18T08:27:15+03:00] INFO: Storing updated cookbooks/mattermost/attributes/default.rb in the cache.
[2025-07-18T08:27:15+03:00] INFO: Storing updated cookbooks/registry/attributes/default.rb in the cache.
  - registry (0.1.0)
[2025-07-18T08:27:15+03:00] INFO: Storing updated cookbooks/monitoring/attributes/default.rb in the cache.
[2025-07-18T08:27:15+03:00] INFO: Storing updated cookbooks/consul/attributes/default.rb in the cache.
  - mattermost (0.1.0)
  - consul (0.1.0)
  - monitoring (0.1.0)
[2025-07-18T08:27:15+03:00] INFO: Storing updated cookbooks/gitaly/recipes/enable.rb in the cache.
[2025-07-18T08:27:15+03:00] INFO: Storing updated cookbooks/praefect/recipes/enable.rb in the cache.
  - praefect (0.1.0)
[2025-07-18T08:27:15+03:00] INFO: Storing updated cookbooks/gitaly/libraries/gitaly.rb in the cache.
[2025-07-18T08:27:15+03:00] INFO: Storing updated cookbooks/gitlab-kas/attributes/default.rb in the cache.
  - gitlab-kas (0.1.0)
  - gitaly (0.1.0)
[2025-07-18T08:27:15+03:00] INFO: Storing updated cookbooks/gitlab-pages/recipes/enable.rb in the cache.
  - letsencrypt (0.1.0)
  - nginx (0.1.0)
  - runit (5.1.7)
  - gitlab-pages (0.1.0)
  - acme (4.1.6)
  - crond (0.1.0)
Installing cookbook gem dependencies:
Compiling cookbooks...
Loading Cinc Auditor profile files:
Loading Cinc Auditor input files:
Loading Cinc Auditor waiver files:
[2025-07-18T08:27:15+03:00] INFO: Generating default secrets
[2025-07-18T08:27:17+03:00] INFO: Generating /etc/gitlab/gitlab-secrets.json file
Converging 4 resources
Recipe: postgresql::bin
  * ruby_block[check_postgresql_version] action run (skipped due to not_if)
  * ruby_block[check_postgresql_version_is_deprecated] action run (skipped due to not_if)
  * ruby_block[Link postgresql bin files to the correct version] action run (skipped due to only_if)
  * template[/opt/gitlab/etc/gitlab-psql-rc] action create (up to date)
[2025-07-18T08:27:17+03:00] INFO: Cinc Client Run complete in 3.503394339 seconds

Running handlers:
[2025-07-18T08:27:17+03:00] INFO: Running report handlers
Running handlers complete
[2025-07-18T08:27:17+03:00] INFO: Report handlers complete
Infra Phase complete, 0/4 resources updated in 05 seconds
Checking PostgreSQL executables: OK
Shutting down all GitLab services except those needed for migrations
ok: down: alertmanager: 0s, normally up
ok: down: gitlab-exporter: 0s, normally up
ok: down: gitlab-kas: 0s, normally up
ok: down: gitlab-workhorse: 0s, normally up
ok: down: logrotate: 1s, normally up
ok: down: nginx: 0s, normally up
ok: down: node-exporter: 0s, normally up
ok: down: postgres-exporter: 0s, normally up
ok: down: prometheus: 0s, normally up
ok: down: puma: 0s, normally up
ok: down: redis-exporter: 0s, normally up
ok: down: registry: 1s, normally up
ok: down: sidekiq: 0s, normally up
Ensuring the required services are running
ok: run: postgresql: (pid 2244) 607172s
ok: run: redis: (pid 2261) 607172s
ok: run: gitaly: (pid 2246) 607172s
run: postgresql: (pid 2244) 607172s; run: log: (pid 2237) 607172s
run: redis: (pid 2261) 607172s; run: log: (pid 2257) 607172s
run: gitaly: (pid 2246) 607172s; run: log: (pid 2242) 607172s
Reconfigure GitLab to apply migrations
[2025-07-18T08:27:44+03:00] INFO: Started Cinc Zero at chefzero://localhost:1 with repository at /opt/gitlab/embedded (One version per cookbook)
Cinc Client, version 18.3.0
Patents: https://www.chef.io/patents
Infra Phase starting
[2025-07-18T08:27:45+03:00] INFO: *** Cinc Client 18.3.0 ***
[2025-07-18T08:27:45+03:00] INFO: Platform: x86_64-linux
[2025-07-18T08:27:45+03:00] INFO: Cinc-client pid: 937649
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/ffi-yajl-2.6.0/lib/ffi_yajl/encoder.rb:42: warning: undefining the allocator of T_DATA class FFI_Yajl::Ext::Encoder::YajlGen
[2025-07-18T08:27:46+03:00] INFO: Setting the run_list to ["recipe[gitlab-base]"] from CLI options
[2025-07-18T08:27:46+03:00] INFO: Run List is [recipe[gitlab-base]]
[2025-07-18T08:27:46+03:00] INFO: Run List expands to [gitlab-base]
[2025-07-18T08:27:46+03:00] INFO: Starting Cinc Client Run for git.company.tld
[2025-07-18T08:27:46+03:00] INFO: Running start handlers
[2025-07-18T08:27:46+03:00] INFO: Start handlers complete.
Resolving cookbooks for run list: ["gitlab-base"]
[2025-07-18T08:27:47+03:00] INFO: Loading cookbooks [gitlab-base@0.1.0, package@0.1.0, gitlab@0.0.1, runit@5.1.7, logrotate@0.1.0, postgresql@0.1.0, redis@0.1.0, monitoring@0.1.0, registry@0.1.0, mattermost@0.1.0, consul@0.1.0, gitaly@0.1.0, praefect@0.1.0, gitlab-kas@0.1.0, gitlab-pages@0.1.0, letsencrypt@0.1.0, nginx@0.1.0, acme@4.1.6, crond@0.1.0]
Synchronizing cookbooks:
[2025-07-18T08:27:47+03:00] INFO: Storing updated cookbooks/gitlab-base/recipes/default.rb in the cache.
[2025-07-18T08:27:47+03:00] INFO: Storing updated cookbooks/gitlab-base/metadata.rb in the cache.
  - gitlab-base (0.1.0)
  - package (0.1.0)
  - runit (5.1.7)
  - logrotate (0.1.0)
  - postgresql (0.1.0)
  - redis (0.1.0)
  - gitlab (0.0.1)
  - monitoring (0.1.0)
  - registry (0.1.0)
  - mattermost (0.1.0)
  - gitaly (0.1.0)
  - praefect (0.1.0)
  - gitlab-pages (0.1.0)
  - letsencrypt (0.1.0)
  - nginx (0.1.0)
  - acme (4.1.6)
  - crond (0.1.0)
  - consul (0.1.0)
  - gitlab-kas (0.1.0)
Installing cookbook gem dependencies:
Compiling cookbooks...
/opt/gitlab/embedded/cookbooks/cache/cookbooks/package/libraries/helpers/selinux_distro_helper.rb:2: warning: already initialized constant SELinuxDistroHelper::REDHAT_RELEASE_FILE
/opt/gitlab/embedded/cookbooks/package/libraries/helpers/selinux_distro_helper.rb:2: warning: previous definition of REDHAT_RELEASE_FILE was here
/opt/gitlab/embedded/cookbooks/cache/cookbooks/package/libraries/helpers/selinux_distro_helper.rb:3: warning: already initialized constant SELinuxDistroHelper::OS_RELEASE_FILE
/opt/gitlab/embedded/cookbooks/package/libraries/helpers/selinux_distro_helper.rb:3: warning: previous definition of OS_RELEASE_FILE was here
/opt/gitlab/embedded/cookbooks/cache/cookbooks/package/libraries/helpers/secrets_helper.rb:4: warning: already initialized constant SecretsHelper::SECRETS_FILE
/opt/gitlab/embedded/cookbooks/package/libraries/helpers/secrets_helper.rb:4: warning: previous definition of SECRETS_FILE was here
/opt/gitlab/embedded/cookbooks/cache/cookbooks/package/libraries/helpers/secrets_helper.rb:5: warning: already initialized constant SecretsHelper::SECRETS_FILE_CHEF_ATTR
/opt/gitlab/embedded/cookbooks/package/libraries/helpers/secrets_helper.rb:5: warning: previous definition of SECRETS_FILE_CHEF_ATTR was here
/opt/gitlab/embedded/cookbooks/cache/cookbooks/package/libraries/helpers/secrets_helper.rb:6: warning: already initialized constant SecretsHelper::SKIP_GENERATE_SECRETS_CHEF_ATTR
/opt/gitlab/embedded/cookbooks/package/libraries/helpers/secrets_helper.rb:6: warning: previous definition of SKIP_GENERATE_SECRETS_CHEF_ATTR was here
/opt/gitlab/embedded/cookbooks/cache/cookbooks/package/libraries/gitlab_cluster.rb:16: warning: already initialized constant GitlabCluster::CONFIG_PATH
/opt/gitlab/embedded/cookbooks/package/libraries/gitlab_cluster.rb:16: warning: previous definition of CONFIG_PATH was here
/opt/gitlab/embedded/cookbooks/cache/cookbooks/package/libraries/gitlab_cluster.rb:17: warning: already initialized constant GitlabCluster::JSON_FILE
/opt/gitlab/embedded/cookbooks/package/libraries/gitlab_cluster.rb:17: warning: previous definition of JSON_FILE was here
Loading Cinc Auditor profile files:
Loading Cinc Auditor input files:
Loading Cinc Auditor waiver files:
[2025-07-18T08:27:47+03:00] WARN: MissingCookbookDependency:
Recipe `gitlab-ee::default` is not in the run_list, and cookbook 'gitlab-ee'
is not a dependency of any cookbook in the run_list. To load this recipe,
first add a dependency of the cookbook 'gitlab-ee' into the metadata
          of the cookbook which depends on 'gitlab-ee'.


================================================================================
Recipe Compile Error in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab-base/recipes/default.rb
================================================================================

Chef::Exceptions::CookbookNotFound
----------------------------------
Cookbook gitlab-ee not found. If you're loading gitlab-ee from another cookbook, make sure you configure the dependency in your metadata

Cookbook Trace: (most recent call first)
----------------------------------------
  /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab-base/recipes/default.rb:4:in `from_file'

Relevant File Content:
----------------------
/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab-base/recipes/default.rb:

  1:  if Dir.exist?(File.absolute_path(File.join(__dir__, "../../gitlab-jh")))
  2:    include_recipe 'gitlab-jh::default'
  3:  elsif Dir.exist?(File.absolute_path(File.join(__dir__, "../../gitlab-ee")))
  4>>   include_recipe 'gitlab-ee::default'
  5:  elsif Dir.exist?(File.absolute_path(File.join(__dir__, "../../gitlab")))
  6:    include_recipe 'gitlab::default'
  7:  end
  8:  

System Info:
------------
chef_version=18.3.0
platform=ubuntu
platform_version=22.04
ruby=ruby 3.2.5 (2024-07-26 revision 31d0f1a2e7) [x86_64-linux]
program_name=/opt/gitlab/embedded/bin/cinc-client
executable=/opt/gitlab/embedded/bin/cinc-client


Running handlers:
[2025-07-18T08:27:47+03:00] ERROR: Running exception handlers
There was an error running gitlab-ctl reconfigure:

Cookbook gitlab-ee not found. If you're loading gitlab-ee from another cookbook, make sure you configure the dependency in your metadata

Running handlers complete
[2025-07-18T08:27:47+03:00] ERROR: Exception handlers complete
Infra Phase failed. 0 resources updated in 02 seconds
[2025-07-18T08:27:47+03:00] FATAL: Stacktrace dumped to /opt/gitlab/embedded/cookbooks/cache/cinc-stacktrace.out
[2025-07-18T08:27:47+03:00] FATAL: ---------------------------------------------------------------------------------------
[2025-07-18T08:27:47+03:00] FATAL: PLEASE PROVIDE THE CONTENTS OF THE stacktrace.out FILE (above) IF YOU FILE A BUG REPORT
[2025-07-18T08:27:47+03:00] FATAL: ---------------------------------------------------------------------------------------
[2025-07-18T08:27:47+03:00] FATAL: Chef::Exceptions::CookbookNotFound: Cookbook gitlab-ee not found. If you're loading gitlab-ee from another cookbook, make sure you configure the dependency in your metadata
===
There was an error running gitlab-ctl reconfigure. Please check the output above for more
details.
===

Here is the stacktrace

Generated at 2025-07-18 08:43:06 +0300
Chef::Exceptions::CookbookNotFound: Cookbook gitlab-ee not found. If you're loading gitlab-ee from another cookbook, make sure you configure the dependency in your metadata
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/chef-18.3.0/lib/chef/cookbook/cookbook_collection.rb:38:in `block in initialize'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/chef-utils-18.3.0/lib/chef-utils/mash.rb:93:in `default'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/chef-utils-18.3.0/lib/chef-utils/mash.rb:93:in `default'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/chef-utils-18.3.0/lib/chef-utils/mash.rb:119:in `[]'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/chef-utils-18.3.0/lib/chef-utils/mash.rb:119:in `[]'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/chef-18.3.0/lib/chef/run_context.rb:428:in `load_recipe'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/chef-18.3.0/lib/chef/run_context.rb:385:in `block in include_recipe'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/chef-18.3.0/lib/chef/run_context.rb:384:in `each'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/chef-18.3.0/lib/chef/run_context.rb:384:in `include_recipe'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/chef-18.3.0/lib/chef/dsl/include_recipe.rb:26:in `include_recipe'
/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab-base/recipes/default.rb:4:in `from_file'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/chef-18.3.0/lib/chef/mixin/from_file.rb:34:in `instance_eval'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/chef-18.3.0/lib/chef/mixin/from_file.rb:34:in `from_file'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/chef-18.3.0/lib/chef/cookbook_version.rb:233:in `load_ruby_recipe'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/chef-18.3.0/lib/chef/cookbook_version.rb:203:in `load_recipe'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/chef-18.3.0/lib/chef/run_context.rb:429:in `load_recipe'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/chef-18.3.0/lib/chef/run_context/cookbook_compiler.rb:228:in `block in compile_recipes'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/chef-18.3.0/lib/chef/run_context/cookbook_compiler.rb:225:in `each'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/chef-18.3.0/lib/chef/run_context/cookbook_compiler.rb:225:in `compile_recipes'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/chef-18.3.0/lib/chef/run_context/cookbook_compiler.rb:109:in `compile'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/chef-18.3.0/lib/chef/run_context.rb:259:in `load'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/chef-18.3.0/lib/chef/policy_builder/expand_node_object.rb:103:in `setup_run_context'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/forwardable-1.3.3/lib/forwardable.rb:240:in `setup_run_context'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/chef-18.3.0/lib/chef/client.rb:499:in `setup_run_context'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/chef-18.3.0/lib/chef/client.rb:294:in `run'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/chef-18.3.0/lib/chef/application.rb:305:in `run_with_graceful_exit_option'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/chef-18.3.0/lib/chef/application.rb:281:in `block in run_chef_client'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/chef-18.3.0/lib/chef/local_mode.rb:42:in `with_server_connectivity'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/chef-18.3.0/lib/chef/application.rb:264:in `run_chef_client'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/chef-18.3.0/lib/chef/application/base.rb:354:in `run_application'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/chef-18.3.0/lib/chef/application.rb:67:in `run'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/chef-bin-18.3.0/bin/cinc-client:25:in `<top (required)>'
/opt/gitlab/embedded/bin/cinc-client:25:in `load'
/opt/gitlab/embedded/bin/cinc-client:25:in `<main>'

Steps to reproduce

I tried to downgrade back to 18.1.2 but it fails because of malformed config file

Malformed configuration JSON file found at /opt/gitlab/embedded/nodes/git.company.tld.json.
This usually happens when your last run of `gitlab-ctl reconfigure` didn't complete successfully.
This file is used to check if any of the unsupported configurations are enabled,
and hence require a working reconfigure before upgrading.

gitlab-ctl reconfigure fails bebause of the missing gitlab-ee cookbook

Configuration

Provide screenshots from the GitLab UI showing relevant configuration, if applicable.
On self-managed instances, add the relevant configuration settings or changes.

Versions

Please add an x whether options apply, and add the version information.

  • Self-managed
  • GitLab.com SaaS
  • Dedicated

Versions

System information
System:         Ubuntu 22.04
Current User:   git
Using RVM:      no
Ruby Version:   3.2.5
Gem Version:    3.6.9
Bundler Version:2.6.5
Rake Version:   13.0.6
Redis Version:  7.2.9
Sidekiq Version:7.3.9
Go Version:     unknown

GitLab information
Version:        18.2.0
Revision:       95351f08aae
Directory:      /opt/gitlab/embedded/service/gitlab-rails
DB Adapter:     PostgreSQL
DB Version:     16.8
URL:            https://git.company.tld
HTTP Clone URL: https://gitcompany.tld/some-group/some-project.git
SSH Clone URL:  git@git.company.tld:some-group/some-project.git
Using LDAP:     yes
Using Omniauth: yes
Omniauth Providers: 

GitLab Shell
Version:        14.43.0
Repository storages:
- default:      unix:/var/opt/gitlab/gitaly/gitaly.socket
GitLab Shell path:              /opt/gitlab/embedded/service/gitlab-shell

Gitaly
- default Address:      unix:/var/opt/gitlab/gitaly/gitaly.socket
- default Version:      18.1.2
- default Git Version:  2.49.0.gl2
2 Likes
2

Same problem here. There are still configuration files left behind from a long time ago when we tried out GitLab EE (AFAIK), maybe that is related?

git.company.tld ~ # dpkg -l | grep gitlab   
iHR gitlab-ce                         18.2.0-ce.0                                 amd64        GitLab Community Edition (including NGINX, Postgres, Redis)
rc  gitlab-ee                         11.9.4-ee.0                                 amd64        GitLab Enterprise Edition (including NGINX, Postgres, Redis)
ii  libgitlab-api-v4-perl             0.26-1                                      all          GitLab API v4 module and command-line client

To make matters worse, the downgrade is broken too, because the upgrade has garbled /opt/gitlab/embedded/nodes/git.company.tld.json:

git.company.tld ~ # apt install gitlab-ce=18.1.2-ce.0                                                  
Paketlisten werden gelesen
 Fertig                                                                                                                                                                      
AbhÀngigkeitsbaum wird aufgebaut
 Fertig                                                            
Statusinformationen werden eingelesen
 Fertig                                                                                                                                                           
Die folgenden Pakete werden durch eine ÄLTERE VERSION ERSETZT (Downgrade):                          
  gitlab-ce                                                                                                                                                                                             
0 aktualisiert, 0 neu installiert, 1 durch eine Àltere Version ersetzt, 0 zu entfernen und 0 nicht aktualisiert.                                                                                        
1 nicht vollstÀndig installiert oder entfernt.                                                                                                                                                          
Es mĂŒssen 1.416 MB an Archiven heruntergeladen werden.                                              
Nach dieser Operation werden 72,3 MB Plattenplatz freigegeben.                                                                                                                                          
Möchten Sie fortfahren? [J/n]                                                                       
Holen:1 https://packages.gitlab.com/gitlab/gitlab-ce/debian bullseye/main amd64 gitlab-ce amd64 18.1.2-ce.0 [1.416 MB]                                                                                  
Es wurden 1.416 MB in 16 s geholt (88,4 MB/s).                                                                                                                                                          
dpkg: Warnung: Version 18.2.0-ce.0 des Paketes gitlab-ce wird durch Àltere Version 18.1.2-ce.0 ersetzt                                                                                                  
(Lese Datenbank ... 200879 Dateien und Verzeichnisse sind derzeit installiert.)                     
Vorbereitung zum Entpacken von .../gitlab-ce_18.1.2-ce.0_amd64.deb ...                                                                                                                                  
Malformed configuration JSON file found at /opt/gitlab/embedded/nodes/git.company.tld.json.            
This usually happens when your last run of `gitlab-ctl reconfigure` didn't complete successfully.                                                                                                       
This file is used to check if any of the unsupported configurations are enabled,                    
and hence require a working reconfigure before upgrading.                                                                                                                                               
Please run `sudo gitlab-ctl reconfigure` to fix it and try again.                                   
dpkg: Fehler beim Bearbeiten des Archivs /var/cache/apt/archives/gitlab-ce_18.1.2-ce.0_amd64.deb (--unpack):                                                                                            
 »neues gitlab-ce-Skript des Paketes pre-installation«-Unterprozess gab den Fehlerwert 1 zurĂŒck                                                                                                         
Fehler traten auf beim Bearbeiten von:                                                                                                                                                                  
 /var/cache/apt/archives/gitlab-ce_18.1.2-ce.0_amd64.deb                                            
E: Sub-process /usr/bin/dpkg returned an error code (1)

The error message says to run gitlab-ctl reconfigure, but that runs into the same error that originally triggered all this. This aspect of the upgrade/downgrade process seems
 suboptimal.

After I recovered the JSON file from our backups, I was able to get our instance running again after downgrading it to 18.1.2-ce.0.

1 Like
3

I also had some configs and files from an old gitlab-ee installation lingering around.

/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab-ee
/var/lib/dpkg/info/gitlab-ee.postrm
/var/lib/dpkg/info/gitlab-ee.list

After removing those and running gitlab-ctl reconfigure the installation managed to finish successfully.

7 Likes
4

This worked for us as well. Thank you very much for the hint.

The files in /var/lib/dpkg/info/ are part of the gitlab-ee package. The proper way to get rid of those is to purge the package, e.g. apt-get purge gitlab-ee. The files /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab-ee require manual deletion.

2 Likes
5

This is 100% caused by the fact that gitlab-ee was removed previously, but the configuration files were left behing. Had the package been purged, then none of the files would have been left behind, and the upgrade would most likely have worked. If anything had been left in the cookbook cache, then obviously the additional manual cleanup as suggested by @KimBrodowski would have fixed that. This is clearly shown in the post by @martinvonwittich from the list of packages:

As you can see the rc gitlab-ee shows that the package has been removed, but configuration files remain. The correct removal procedure would have been:

apt-get purge gitlab-ee

as the package had already been removed, then the purge command can be issued to remove the remnants left behind as shown in the previous post by Kim. This is not unique to Gitlab, but plenty of packages removed on Debian/Ubuntu leave configuration files behind.

I have a clean gitlab-ce install which has never had gitlab-ee installed, and the upgrade does not suffer any of these symptoms whatsoever. The upgrade from 18.1.2 to 18.2.0 occurred without any problems at all. Had gitlab-ee not been installed on these systems previously, they too would not have experienced the problem.

Just to clarify :slight_smile:

It is interesting though that if previous upgrades have been completed without issue, why it’s only started to appear now :thinking: so definitely weird.

1 Like
6

Thanks a lot. Fixed the issue for me too.

7

I encountered this error on a machine with the opposite case: CE was installed (but not purged), while EE is currently installed.

@iwalker

The correct removal procedure would have been

Note that the official documentation does not instruct users to purge.

2 Likes
8

They won’t suggest purge by default, because it deletes your config files. But any Linux administrator should know to use purge if attempting to use alternative packages and changing between gitlab-ce and gitlab-ee or any other app in fact that leaves config files behind.

9

Sure! Let’s get rid of the entire documentation while we’re at it, because “administrators should know.”

First-time users have no knowledge of GitLab internals. As far as they’re concerned, gitlab-ee might place the same files as gitlab-ce, which would make purging not only useless but problematic.

If users need to purge, the documentation should tell them to. Relying on users to ‘know’ is bad documentation.

1 Like
10

You are missing the point. Purging a package is the package manager on the system be it dnf, apt, or whatever. That’s not Gitlab’s problem. If you are using Linux, then you should be learning how to use the package manager, how to remove packages, install them, etc. That’s not Gitlab’s problem.

Purging a package isn’t gitlab internals. Thats the responsibility of the package manager. Gitlab isn’t here to teach you how to be a Linux admin. Plenty of howtos on how to use dnf, apt on the internet. No different than knowing how to remove a Windows app.

1 Like
11

Thanks, the cached gitlab-ee cookbook files seem to be the issue. I removed those & restored the /opt/gitlab/embedded/nodes/git.company.tld.json. file mentioned in the thread as it was nearly empty and then was able to re-run gitlab-ctl reconfigure & gitlab-ctl restart to get things back up and running

1 Like
12

Gitlab 18.2.0 > 18.2.1 community on AlmaLinux 9 upgrade here which failed the same way, similar steps to recover for the benefit of anyone else:
sudo rm /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab-ee -rf
restore from backup /opt/gitlab/embedded/nodes/my.host.name.json
sudo gitlab-ctl reconfigure (this failed reporting a Postgres username error)
sudo touch /etc/gitlab/gitlab.rb
sudo gitlab-ctl reconfigure (worked this time, not entirely sure why)
sudo gitlab-ctl restart
The restart command completed but the web UI was still returning HTTP 502’s for several minutes before it started working just as I was about to dig further into the backups.

13

Same, for the record, this doesn’t have to be caused by trying out EE, in the past there was some weak dependency declarations where you could upgrade to EE by mistake.

I think I ran into it multiple times apparently

[root@gitlab ~]# ls -l /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab-ee
total 0
drwxr-xr-x. 2 root root  6 Dec  3  2022 attributes
drwxr-xr-x. 2 root root  6 Jun 25  2020 definitions
drwxr-xr-x. 2 root root  6 Dec  3  2022 libraries
drwxr-xr-x. 2 root root  6 Dec  3  2022 recipes
drwxr-xr-x. 2 root root  6 Dec  3  2022 resources
drwxr-xr-x. 3 root root 21 Nov  5  2018 templates

To be clear, the fix here should be to not execute cookbooks from an edition that is known to be wrong. Don’t have chef iterate over anything that looks like it’s still good, instead qualify the things it runs. for obvious security reasons, too.

14

quoting:
“Purging a package isn’t gitlab internals. Thats the responsibility of the package manager. Gitlab isn’t here to teach you how to be a Linux admin. Plenty of howtos on how to use dnf, apt on the internet. No different than knowing how to remove a Windows app.”

(the quoting got broken for some reason, I’ve just added that back once more I hope it works now.)

I would suggest you hold your breath there.
The package manager executes what the creator of the package has told it to do.
A user is intended to abide by those choices and cleanup tasks belong in a corresponding documenation that should be pointed at - but in the case of the invention of omnibus installers the responsibility for what happens in the process of installation or uninstallation is solely NOT of the user('s sysadmin skills). That was the whole marketing point of the omnibus installer and it’s pretty wild to then turn around and hang it on them. You got a choice if you wanna be that person.
Gitlab has various little race conditions here and in -runner where it will nuke its own configuration and if i may say, that is also not the user’s problem but something where one needs to have a good clearheaded day and work on the cookbooks. not trivial but a fact of life.
no config management policy should fail ‘unsafe’ and tear down the configuration it needs to climb back out of a pit.

I mean, did you LOOK at the path here? /opt/gitlab/embedded
This is very much not the territory of “user should prune”. Embedded literally emphasizes these are internal components that the package will maintain.

Maybe you could phrase it like this: In gitlab there’s a few rough edges with the uninstall in such situations, hopping between CE and EE and back due to install errors or bugs is not an intended scenario and this can cause a few troubles. If one is savvy with the usual package managers, it’s easier to clear up such things by recognizing which files are leftovers.
In the case of omnibus installs, one option is to always start from a clean slate and to look which files were modified by gitlab-ctl post-install, so one gets a better understanding of what the gitlab install actually consists of - this will also help in such cases in the future.

Whatever you would like to say - maybe try like that?

15

Purging a package isn’t gitlab internals. Thats the responsibility of the package manager. Gitlab isn’t here to teach you how to be a Linux admin. Plenty of howtos on how to use dnf, apt on the internet. No different than knowing how to remove a Windows app.

@Darkfader You’re mis-attributing that quote to me. I was not the author of it

16

i’m sorry, I don’t know why I got that so wrong. Will try again.
Seems bugged somehow. Let me kill my cache. it seems it somehow has an off-by-one error on the quotes. I’ve fixed the main wrong attribution and gonna let it be for the night since either I’ve suddenly forgotten how to click on reply or it’s bugged.