Update 18.1.2 -> 18.1.3/4 always fails with error at reconfiguration

Update from 18.1.2 to 18.1.3/4 fails with an access error

I have a long-running instance of GitLab-omnibus (5 years+) with the current state:

root@gitlab ~ # gitlab-rake gitlab:env:info

System information
System:         Debian 12
Proxy:          no
Current User:   git
Using RVM:      no
Ruby Version:   3.2.5
Gem Version:    3.6.9
Bundler Version:2.6.9
Rake Version:   13.0.6
Redis Version:  7.2.9
Sidekiq Version:7.3.9
Go Version:     unknown

GitLab information
Version:        18.1.2-ee
Revision:       cd9f1ac5ba9
Directory:      /opt/gitlab/embedded/service/gitlab-rails
DB Adapter:     PostgreSQL
DB Version:     16.8
URL:            https://gitlab.my.services
HTTP Clone URL: https://gitlab.my.services/some-group/some-project.git
SSH Clone URL:  git@gitlab.my.services:some-group/some-project.git
Elasticsearch:  no
Geo:            no
Using LDAP:     yes
Using Omniauth: yes
Omniauth Providers:

GitLab Shell
Version:        14.42.0
Repository storages:
- default:      unix:/var/opt/gitlab/gitaly/gitaly.socket
GitLab Shell path:              /opt/gitlab/embedded/service/gitlab-shell

Gitaly
- default Address:      unix:/var/opt/gitlab/gitaly/gitaly.socket
- default Version:      18.1.2
- default Git Version:  2.49.0.gl2
gitlab-rake gitlab:env:info  21.34s user 5.37s system 86% cpu 30.859 total

root@gitlab ~ # gitlab-ctl status
run: alertmanager: (pid 1294) 404s; run: log: (pid 1287) 404s
run: gitaly: (pid 1289) 404s; run: log: (pid 1283) 404s
run: gitlab-exporter: (pid 1282) 404s; run: log: (pid 1278) 404s
run: gitlab-kas: (pid 1273) 404s; run: log: (pid 1267) 404s
run: gitlab-workhorse: (pid 1262) 404s; run: log: (pid 1259) 404s
run: logrotate: (pid 1263) 404s; run: log: (pid 1258) 404s
run: nginx: (pid 1256) 404s; run: log: (pid 1249) 404s
run: node-exporter: (pid 1293) 405s; run: log: (pid 1288) 405s
run: postgres-exporter: (pid 1291) 405s; run: log: (pid 1286) 405s
run: postgresql: (pid 1279) 405s; run: log: (pid 1261) 405s
run: prometheus: (pid 1299) 405s; run: log: (pid 1296) 405s
run: puma: (pid 1297) 405s; run: log: (pid 1295) 405s
run: redis: (pid 1276) 405s; run: log: (pid 1260) 405s
run: redis-exporter: (pid 1265) 405s; run: log: (pid 1257) 405s
run: registry: (pid 1303) 405s; run: log: (pid 1300) 405s
run: sidekiq: (pid 1292) 405s; run: log: (pid 1285) 405s

As always, I try to do a linear update:

root@gitlab # apt install gitlab-ee=18.1.4-ee.0

Installation is running, but reconfiguration always fails at the following position

  * database_objects[postgresql] action create
    * postgresql_user[gitlab] action create
      * execute[create gitlab postgresql user] action run

        ================================================================================
        Error executing action `run` on resource 'execute[create gitlab postgresql user]'
        ================================================================================

        Errno::EACCES
        -------------
        Permission denied - /opt/gitlab/bin/gitlab-psql -d template1 -c "CREATE USER \"gitlab\""

        Cookbook Trace: (most recent call first)
        ----------------------------------------
        /opt/gitlab/embedded/cookbooks/cache/cookbooks/postgresql/resources/user.rb:13:in `block in class_from_file'
        /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/database_objects.rb:28:in `block in class_from_file'

        Resource Declaration:
        ---------------------
        # In /opt/gitlab/embedded/cookbooks/cache/cookbooks/postgresql/resources/user.rb

         13:   execute "create #{new_resource.username} postgresql user" do
         14:     command %(/opt/gitlab/bin/#{new_resource.helper.service_cmd} -d template1 -c "#{query}")
         15:     user account_helper.postgresql_user
         16:     only_if { new_resource.helper.is_running? && new_resource.helper.is_ready? }
         17:     not_if { new_resource.helper.is_offline_or_readonly? || new_resource.helper.user_exists?(new_resource.username) }
         18:   end
         19:

        Compiled Resource:
        ------------------
        # Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/postgresql/resources/user.rb:13:in `block in class_from_file'

        execute("create gitlab postgresql user") do
          action [:run]
          default_guard_interpreter :execute
          command "/opt/gitlab/bin/gitlab-psql -d template1 -c \"CREATE USER \\\"gitlab\\\"\""
          declared_type :execute
          cookbook_name "postgresql"
          recipe_name "standalone"
          user "gitlab-psql"
          not_if { #code block }
          only_if { #code block }
        end

        System Info:
        ------------
        chef_version=18.3.0
        platform=debian
        platform_version=12
        ruby=ruby 3.2.5 (2024-07-26 revision 31d0f1a2e7) [x86_64-linux]
        program_name=/opt/gitlab/embedded/bin/cinc-client
        executable=/opt/gitlab/embedded/bin/cinc-client


      ================================================================================
      Error executing action `create` on resource 'postgresql_user[gitlab]'
      ================================================================================

      Errno::EACCES
      -------------
      execute[create gitlab postgresql user] (postgresql::standalone line 13) had an error: Errno::EACCES: Permission denied - /opt/gitlab/bin/gitlab-psql -d template1 -c "CREATE USER \"gitlab\""

      Cookbook Trace: (most recent call first)
      ----------------------------------------
      /opt/gitlab/embedded/cookbooks/cache/cookbooks/postgresql/resources/user.rb:13:in `block in class_from_file'
      /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/database_objects.rb:28:in `block in class_from_file'

      Resource Declaration:
      ---------------------
      # In /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/database_objects.rb

       28:   postgresql_user gitlab_sql_user do
       29:     password "md5#{gitlab_sql_user_password}" unless gitlab_sql_user_password.nil?
       30:     action :create
       31:   end
       32:

      Compiled Resource:
      ------------------
      # Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/database_objects.rb:28:in `block in class_from_file'

      postgresql_user("gitlab") do
        action [:create]
        default_guard_interpreter :default
        declared_type :postgresql_user
        cookbook_name "postgresql"
        recipe_name "standalone"
        helper "*sensitive value suppressed*"
      end

      System Info:
      ------------
      chef_version=18.3.0
      platform=debian
      platform_version=12
      ruby=ruby 3.2.5 (2024-07-26 revision 31d0f1a2e7) [x86_64-linux]
      program_name=/opt/gitlab/embedded/bin/cinc-client
      executable=/opt/gitlab/embedded/bin/cinc-client


    ================================================================================
    Error executing action `create` on resource 'database_objects[postgresql]'
    ================================================================================

    Errno::EACCES
    -------------
    postgresql_user[gitlab] (postgresql::standalone line 28) had an error: Errno::EACCES: execute[create gitlab postgresql user] (postgresql::standalone line 13) had an error: Errno::EACCES: Permission denied - /opt/gitlab/bin/gitlab-psql -d template1 -c "CREATE USER \"gitlab\""

    Cookbook Trace: (most recent call first)
    ----------------------------------------
    /opt/gitlab/embedded/cookbooks/cache/cookbooks/postgresql/resources/user.rb:13:in `block in class_from_file'
    /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/database_objects.rb:28:in `block in class_from_file'

    Resource Declaration:
    ---------------------
    # In /opt/gitlab/embedded/cookbooks/cache/cookbooks/postgresql/recipes/standalone.rb

     32: database_objects 'postgresql' do
     33:   pg_helper pg_helper
     34:   account_helper account_helper
     35:   not_if { pg_helper.replica? }
     36: end
     37:

    Compiled Resource:
    ------------------
    # Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/postgresql/recipes/standalone.rb:32:in `from_file'

    database_objects("postgresql") do
      action [:create]
      default_guard_interpreter :default
      declared_type :database_objects
      cookbook_name "postgresql"
      recipe_name "standalone"
      pg_helper "*sensitive value suppressed*"
      account_helper "*sensitive value suppressed*"
      not_if { #code block }
    end

    System Info:
    ------------
    chef_version=18.3.0
    platform=debian
    platform_version=12
    ruby=ruby 3.2.5 (2024-07-26 revision 31d0f1a2e7) [x86_64-linux]
    program_name=/opt/gitlab/embedded/bin/cinc-client
    executable=/opt/gitlab/embedded/bin/cinc-client

[2025-08-19T22:36:41+02:00] INFO: Running queued delayed notifications before re-raising exception
[2025-08-19T22:36:41+02:00] INFO: version_file[Create version file for Gitaly] sending hup action to runit_service[gitaly] (delayed)
Recipe: gitaly::enable
  * runit_service[gitaly] action hup[2025-08-19T22:36:41+02:00] INFO: runit_service[gitaly] signalled (HUP)
[2025-08-19T22:36:41+02:00] INFO: runit_service[gitaly] sent hup

    - send hup to runit_service[gitaly]

Running handlers:
[2025-08-19T22:36:41+02:00] ERROR: Running exception handlers
There was an error running gitlab-ctl reconfigure:

database_objects[postgresql] (postgresql::standalone line 32) had an error: Errno::EACCES: postgresql_user[gitlab] (postgresql::standalone line 28) had an error: Errno::EACCES: execute[create gitlab postgresql user] (postgresql::standalone line 13) had an error: Errno::EACCES: Permission denied - /opt/gitlab/bin/gitlab-psql -d template1 -c "CREATE USER \"gitlab\""

Running handlers complete
[2025-08-19T22:36:41+02:00] ERROR: Exception handlers complete
Infra Phase failed. 12 resources updated in 13 seconds
[2025-08-19T22:36:41+02:00] FATAL: Stacktrace dumped to /opt/gitlab/embedded/cookbooks/cache/cinc-stacktrace.out
[2025-08-19T22:36:41+02:00] FATAL: ---------------------------------------------------------------------------------------
[2025-08-19T22:36:41+02:00] FATAL: PLEASE PROVIDE THE CONTENTS OF THE stacktrace.out FILE (above) IF YOU FILE A BUG REPORT
[2025-08-19T22:36:41+02:00] FATAL: ---------------------------------------------------------------------------------------
[2025-08-19T22:36:41+02:00] FATAL: Errno::EACCES: database_objects[postgresql] (postgresql::standalone line 32) had an error: Errno::EACCES: postgresql_user[gitlab] (postgresql::standalone line 28) had an error: Errno::EACCES: execute[create gitlab postgresql user] (postgresql::standalone line 13) had an error: Errno::EACCES: Permission denied - /opt/gitlab/bin/gitlab-psql -d template1 -c "CREATE USER \"gitlab\""
===
There was an error running gitlab-ctl reconfigure. Please check the output above for more
details.
===

dpkg: error processing package gitlab-ee (--configure):
 installed gitlab-ee package post-installation script subprocess returned error exit status 1
Errors were encountered while processing:
 gitlab-ee
Error: GDBus.Error:org.freedesktop.DBus.Error.ServiceUnknown: The name org.freedesktop.PackageKit was not provided by any .service files
E: Sub-process /usr/bin/dpkg returned an error code (1)
apt install gitlab-ee=18.1.4-ee.0  205.86s user 56.93s system 108% cpu 4:01.43 total

As always, I try to do a linear update:

  * database_objects[postgresql] action create
    * postgresql_user[gitlab] action create
      * execute[create gitlab postgresql user] action run

        ================================================================================
        Error executing action `run` on resource 'execute[create gitlab postgresql user]'
        ================================================================================

        Errno::EACCES
        -------------
        Permission denied - /opt/gitlab/bin/gitlab-psql -d template1 -c "CREATE USER \"gitlab\""

        Cookbook Trace: (most recent call first)
        ----------------------------------------
        /opt/gitlab/embedded/cookbooks/cache/cookbooks/postgresql/resources/user.rb:13:in `block in class_from_file'
        /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/database_objects.rb:28:in `block in class_from_file'

        Resource Declaration:
        ---------------------
        # In /opt/gitlab/embedded/cookbooks/cache/cookbooks/postgresql/resources/user.rb

         13:   execute "create #{new_resource.username} postgresql user" do
         14:     command %(/opt/gitlab/bin/#{new_resource.helper.service_cmd} -d template1 -c "#{query}")
         15:     user account_helper.postgresql_user
         16:     only_if { new_resource.helper.is_running? && new_resource.helper.is_ready? }
         17:     not_if { new_resource.helper.is_offline_or_readonly? || new_resource.helper.user_exists?(new_resource.username) }
         18:   end
         19:

        Compiled Resource:
        ------------------
        # Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/postgresql/resources/user.rb:13:in `block in class_from_file'

        execute("create gitlab postgresql user") do
          action [:run]
          default_guard_interpreter :execute
          command "/opt/gitlab/bin/gitlab-psql -d template1 -c \"CREATE USER \\\"gitlab\\\"\""
          declared_type :execute
          cookbook_name "postgresql"
          recipe_name "standalone"
          user "gitlab-psql"
          not_if { #code block }
          only_if { #code block }
        end

        System Info:
        ------------
        chef_version=18.3.0
        platform=debian
        platform_version=12
        ruby=ruby 3.2.5 (2024-07-26 revision 31d0f1a2e7) [x86_64-linux]
        program_name=/opt/gitlab/embedded/bin/cinc-client
        executable=/opt/gitlab/embedded/bin/cinc-client


      ================================================================================
      Error executing action `create` on resource 'postgresql_user[gitlab]'
      ================================================================================

      Errno::EACCES
      -------------
      execute[create gitlab postgresql user] (postgresql::standalone line 13) had an error: Errno::EACCES: Permission denied - /opt/gitlab/bin/gitlab-psql -d template1 -c "CREATE USER \"gitlab\""

      Cookbook Trace: (most recent call first)
      ----------------------------------------
      /opt/gitlab/embedded/cookbooks/cache/cookbooks/postgresql/resources/user.rb:13:in `block in class_from_file'
      /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/database_objects.rb:28:in `block in class_from_file'

      Resource Declaration:
      ---------------------
      # In /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/database_objects.rb

       28:   postgresql_user gitlab_sql_user do
       29:     password "md5#{gitlab_sql_user_password}" unless gitlab_sql_user_password.nil?
       30:     action :create
       31:   end
       32:

      Compiled Resource:
      ------------------
      # Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/database_objects.rb:28:in `block in class_from_file'

      postgresql_user("gitlab") do
        action [:create]
        default_guard_interpreter :default
        declared_type :postgresql_user
        cookbook_name "postgresql"
        recipe_name "standalone"
        helper "*sensitive value suppressed*"
      end

      System Info:
      ------------
      chef_version=18.3.0
      platform=debian
      platform_version=12
      ruby=ruby 3.2.5 (2024-07-26 revision 31d0f1a2e7) [x86_64-linux]
      program_name=/opt/gitlab/embedded/bin/cinc-client
      executable=/opt/gitlab/embedded/bin/cinc-client


    ================================================================================
    Error executing action `create` on resource 'database_objects[postgresql]'
    ================================================================================

    Errno::EACCES
    -------------
    postgresql_user[gitlab] (postgresql::standalone line 28) had an error: Errno::EACCES: execute[create gitlab postgresql user] (postgresql::standalone line 13) had an error: Errno::EACCES: Permission denied - /opt/gitlab/bin/gitlab-psql -d template1 -c "CREATE USER \"gitlab\""

    Cookbook Trace: (most recent call first)
    ----------------------------------------
    /opt/gitlab/embedded/cookbooks/cache/cookbooks/postgresql/resources/user.rb:13:in `block in class_from_file'
    /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/database_objects.rb:28:in `block in class_from_file'

    Resource Declaration:
    ---------------------
    # In /opt/gitlab/embedded/cookbooks/cache/cookbooks/postgresql/recipes/standalone.rb

     32: database_objects 'postgresql' do
     33:   pg_helper pg_helper
     34:   account_helper account_helper
     35:   not_if { pg_helper.replica? }
     36: end
     37:

    Compiled Resource:
    ------------------
    # Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/postgresql/recipes/standalone.rb:32:in `from_file'

    database_objects("postgresql") do
      action [:create]
      default_guard_interpreter :default
      declared_type :database_objects
      cookbook_name "postgresql"
      recipe_name "standalone"
      pg_helper "*sensitive value suppressed*"
      account_helper "*sensitive value suppressed*"
      not_if { #code block }
    end

    System Info:
    ------------
    chef_version=18.3.0
    platform=debian
    platform_version=12
    ruby=ruby 3.2.5 (2024-07-26 revision 31d0f1a2e7) [x86_64-linux]
    program_name=/opt/gitlab/embedded/bin/cinc-client
    executable=/opt/gitlab/embedded/bin/cinc-client

[2025-08-19T22:36:41+02:00] INFO: Running queued delayed notifications before re-raising exception
[2025-08-19T22:36:41+02:00] INFO: version_file[Create version file for Gitaly] sending hup action to runit_service[gitaly] (delayed)
Recipe: gitaly::enable
  * runit_service[gitaly] action hup[2025-08-19T22:36:41+02:00] INFO: runit_service[gitaly] signalled (HUP)
[2025-08-19T22:36:41+02:00] INFO: runit_service[gitaly] sent hup

    - send hup to runit_service[gitaly]

Running handlers:
[2025-08-19T22:36:41+02:00] ERROR: Running exception handlers
There was an error running gitlab-ctl reconfigure:

database_objects[postgresql] (postgresql::standalone line 32) had an error: Errno::EACCES: postgresql_user[gitlab] (postgresql::standalone line 28) had an error: Errno::EACCES: execute[create gitlab postgresql user] (postgresql::standalone line 13) had an error: Errno::EACCES: Permission denied - /opt/gitlab/bin/gitlab-psql -d template1 -c "CREATE USER \"gitlab\""

Running handlers complete
[2025-08-19T22:36:41+02:00] ERROR: Exception handlers complete
Infra Phase failed. 12 resources updated in 13 seconds
[2025-08-19T22:36:41+02:00] FATAL: Stacktrace dumped to /opt/gitlab/embedded/cookbooks/cache/cinc-stacktrace.out
[2025-08-19T22:36:41+02:00] FATAL: ---------------------------------------------------------------------------------------
[2025-08-19T22:36:41+02:00] FATAL: PLEASE PROVIDE THE CONTENTS OF THE stacktrace.out FILE (above) IF YOU FILE A BUG REPORT
[2025-08-19T22:36:41+02:00] FATAL: ---------------------------------------------------------------------------------------
[2025-08-19T22:36:41+02:00] FATAL: Errno::EACCES: database_objects[postgresql] (postgresql::standalone line 32) had an error: Errno::EACCES: postgresql_user[gitlab] (postgresql::standalone line 28) had an error: Errno::EACCES: execute[create gitlab postgresql user] (postgresql::standalone line 13) had an error: Errno::EACCES: Permission denied - /opt/gitlab/bin/gitlab-psql -d template1 -c "CREATE USER \"gitlab\""
===
There was an error running gitlab-ctl reconfigure. Please check the output above for more
details.
===

dpkg: error processing package gitlab-ee (--configure):
 installed gitlab-ee package post-installation script subprocess returned error exit status 1
Errors were encountered while processing:
 gitlab-ee
Error: GDBus.Error:org.freedesktop.DBus.Error.ServiceUnknown: The name org.freedesktop.PackageKit was not provided by any .service files
E: Sub-process /usr/bin/dpkg returned an error code (1)
apt install gitlab-ee=18.1.4-ee.0  205.86s user 56.93s system 108% cpu 4:01.43 total

The stacktrace file:

root@gitlab ~ # cat /opt/gitlab/embedded/cookbooks/cache/cinc-stacktrace.out
Generated at 2025-08-19 22:36:41 +0200
Errno::EACCES: database_objects[postgresql] (postgresql::standalone line 32) had an error: Errno::EACCES: postgresql_user[gitlab] (postgresql::standalone line 28) had an error: Errno::EACCES: execute[create gitlab postgresql user] (postgresql::standalone line 13) had an error: Errno::EACCES: Permission denied - /opt/gitlab/bin/gitlab-psql -d template1 -c "CREATE USER \"gitlab\""
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/mixlib-shellout-3.2.8/lib/mixlib/shellout/unix.rb:343:in `exec'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/mixlib-shellout-3.2.8/lib/mixlib/shellout/unix.rb:343:in `block in fork_subprocess'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/mixlib-shellout-3.2.8/lib/mixlib/shellout/unix.rb:321:in `fork'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/mixlib-shellout-3.2.8/lib/mixlib/shellout/unix.rb:321:in `fork_subprocess'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/mixlib-shellout-3.2.8/lib/mixlib/shellout/unix.rb:97:in `run_command'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/mixlib-shellout-3.2.8/lib/mixlib/shellout.rb:270:in `run_command'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/mixlib-shellout-3.2.8/lib/mixlib/shellout/helper.rb:154:in `__shell_out_command'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/mixlib-shellout-3.2.8/lib/mixlib/shellout/helper.rb:128:in `shell_out_compacted!'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/mixlib-shellout-3.2.8/lib/mixlib/shellout/helper.rb:54:in `shell_out!'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/chef-18.3.0/lib/chef/provider/execute.rb:52:in `block (2 levels) in <class:Execute>'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/chef-18.3.0/lib/chef/mixin/why_run.rb:51:in `add_action'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/chef-18.3.0/lib/chef/provider.rb:293:in `converge_by'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/chef-18.3.0/lib/chef/provider/execute.rb:50:in `block in <class:Execute>'
(eval):2:in `block in action_run'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/chef-18.3.0/lib/chef/provider.rb:304:in `instance_eval'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/chef-18.3.0/lib/chef/provider.rb:304:in `compile_and_converge_action'
(eval):2:in `action_run'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/chef-18.3.0/lib/chef/provider.rb:245:in `run_action'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/chef-18.3.0/lib/chef/resource.rb:601:in `block in run_action'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/chef-18.3.0/lib/chef/resource.rb:628:in `with_umask'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/chef-18.3.0/lib/chef/resource.rb:600:in `run_action'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/chef-18.3.0/lib/chef/runner.rb:74:in `run_action'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/chef-18.3.0/lib/chef/runner.rb:108:in `block in run_all_actions'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/chef-18.3.0/lib/chef/runner.rb:108:in `each'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/chef-18.3.0/lib/chef/runner.rb:108:in `run_all_actions'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/chef-18.3.0/lib/chef/resource_collection.rb:64:in `insert'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/chef-18.3.0/lib/chef/dsl/declare_resource.rb:267:in `declare_resource'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/chef-18.3.0/lib/chef/dsl/resources.rb:36:in `execute'
/opt/gitlab/embedded/cookbooks/cache/cookbooks/postgresql/resources/user.rb:13:in `block in class_from_file'
(eval):2:in `block in action_create'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/chef-18.3.0/lib/chef/provider.rb:304:in `instance_eval'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/chef-18.3.0/lib/chef/provider.rb:304:in `compile_and_converge_action'
(eval):2:in `action_create'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/chef-18.3.0/lib/chef/provider.rb:245:in `run_action'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/chef-18.3.0/lib/chef/resource.rb:601:in `block in run_action'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/chef-18.3.0/lib/chef/resource.rb:628:in `with_umask'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/chef-18.3.0/lib/chef/resource.rb:600:in `run_action'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/chef-18.3.0/lib/chef/runner.rb:74:in `run_action'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/chef-18.3.0/lib/chef/runner.rb:108:in `block in run_all_actions'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/chef-18.3.0/lib/chef/runner.rb:108:in `each'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/chef-18.3.0/lib/chef/runner.rb:108:in `run_all_actions'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/chef-18.3.0/lib/chef/resource_collection.rb:64:in `insert'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/chef-18.3.0/lib/chef/dsl/declare_resource.rb:267:in `declare_resource'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/chef-18.3.0/lib/chef/dsl/resources.rb:36:in `postgresql_user'
/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/database_objects.rb:28:in `block in class_from_file'
(eval):2:in `block in action_create'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/chef-18.3.0/lib/chef/provider.rb:304:in `instance_eval'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/chef-18.3.0/lib/chef/provider.rb:304:in `compile_and_converge_action'
(eval):2:in `action_create'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/chef-18.3.0/lib/chef/provider.rb:245:in `run_action'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/chef-18.3.0/lib/chef/resource.rb:601:in `block in run_action'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/chef-18.3.0/lib/chef/resource.rb:628:in `with_umask'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/chef-18.3.0/lib/chef/resource.rb:600:in `run_action'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/chef-18.3.0/lib/chef/runner.rb:74:in `run_action'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/chef-18.3.0/lib/chef/runner.rb:108:in `block in run_all_actions'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/chef-18.3.0/lib/chef/runner.rb:108:in `each'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/chef-18.3.0/lib/chef/runner.rb:108:in `run_all_actions'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/chef-18.3.0/lib/chef/runner.rb:132:in `block in converge'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/chef-18.3.0/lib/chef/resource_collection/resource_list.rb:96:in `block in execute_each_resource'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/chef-18.3.0/lib/chef/resource_collection/stepable_iterator.rb:114:in `call_iterator_block'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/chef-18.3.0/lib/chef/resource_collection/stepable_iterator.rb:85:in `step'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/chef-18.3.0/lib/chef/resource_collection/stepable_iterator.rb:103:in `iterate'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/chef-18.3.0/lib/chef/resource_collection/stepable_iterator.rb:54:in `each_with_index'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/chef-18.3.0/lib/chef/resource_collection/resource_list.rb:94:in `execute_each_resource'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/forwardable-1.3.3/lib/forwardable.rb:240:in `execute_each_resource'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/chef-18.3.0/lib/chef/runner.rb:130:in `converge'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/chef-18.3.0/lib/chef/client.rb:860:in `block in converge'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/chef-18.3.0/lib/chef/client.rb:855:in `catch'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/chef-18.3.0/lib/chef/client.rb:855:in `converge'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/chef-18.3.0/lib/chef/client.rb:879:in `converge_and_save'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/chef-18.3.0/lib/chef/client.rb:298:in `run'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/chef-18.3.0/lib/chef/application.rb:305:in `run_with_graceful_exit_option'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/chef-18.3.0/lib/chef/application.rb:281:in `block in run_chef_client'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/chef-18.3.0/lib/chef/local_mode.rb:42:in `with_server_connectivity'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/chef-18.3.0/lib/chef/application.rb:264:in `run_chef_client'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/chef-18.3.0/lib/chef/application/base.rb:354:in `run_application'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/chef-18.3.0/lib/chef/application.rb:67:in `run'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/chef-bin-18.3.0/bin/cinc-client:25:in `<top (required)>'
/opt/gitlab/embedded/bin/cinc-client:25:in `load'
/opt/gitlab/embedded/bin/cinc-client:25:in `<main>'#   

In this state, there is no way to start anything related to the progress user (which is possible before starting the update)

All the called binaries/scripts are available and do have the proper rights. There are no noexec or se-linux, or mount locks active. It appears that running the setup or configuration breaks that configuration “somehow”, and I always need to revert back to the snapshot I took.

I have no idea what to do next, because it seems I’m stuck with the current version of GitLab (or start from scratch with a new instance, which I would prefer not to do).

I would appreciate it if someone could help me get the updates running again!

According to the upgrade path here: Upgrade Path you can go direct to 18.2.4. There could well be particular problems with the 18.1.3 and 18.1.4 releases.

I would suggest missing them, since they are not important updates that include database migrations there is no need to upgrade to them first. Obviously make sure you have a backup/snapshot of your machine to revert if the upgrade process doesn’t work - considering the current problems you are having as well.

Update to 18.2.4 didn’t work too

BUT the root cause of the problem was a wrong decision of my update sequence!

apt update, did show some additional updates along with gitlab-ee, so I decided to update gitlab before the other packages.

This time, I tried to update all the other packages first:

apt install ca-certificates docker-buildx-plugin docker-ce-cli docker-ce-rootless-extras docker-ce docker-compose-plugin (and some more)

I did a reboot after all packages were installed, and waited for GitLab 18.1.2 to run again.
I then performed the GitLab update (direct to 18.2.4), and this time it ran smoothly.

So be aware that updating GitLab MAY FAIL if other updates are pending (even though I’m not sure why)!

Possibly because gitlab updated the dependencies for their packages to use the newer versions rather than the old one - although if that was the case installing the new gitlab package would have pulled them in anyway. It would be wise to ensure the entire system is up-to-date on a regular basis anyway, rather than just selectively upgrading packages.

Yes, strangely enough, because unattended-upgrades is running, and it updates everything, except GitLab. So I hardly remember when such a list of packages needed manual attention (the system has been running for years now, and even went from Debian 11 to Debian 12 meanwhile).

Uff, I hate that package - I always remove unattended-upgrades. I prefer to control when updates happen on my system, at least I know then if I do something and something stops working it can be because of that. With updates installing in the background without me potentially knowing, it’d take longer to figure out what could have introduced the problem.

That, and also when I log into my system to install something, and then find I can’t because unattended-upgrades has started an apt process that I have to wait for it to finish.

I suppose some people might find it useful, it’s just something that I absolutely abhor