Update self-hosted gitlab container from 16.11.0 to 16.11.1

Hello everyone

i wanted to update my self-hosted gitlab container from 16.11.0 to 16.11.1. unfortunately the container keeps crashing and the log gives me this output:

Recipe: letsencrypt::http_authorization
  * letsencrypt_certificate[gitlab.mydomain.de] action create
    * acme_certificate[staging] action create
      * file[gitlab.mydomain.de SSL key] action nothing (skipped due to action :nothing)
      * file[gitlab.mydomain.de SSL key] action create_if_missing (up to date)
opening connection to acme-staging-v02.api.letsencrypt.org:443...
opened
starting SSL for acme-staging-v02.api.letsencrypt.org:443...
SSL established, protocol: TLSv1.3, cipher: TLS_AES_256_GCM_SHA384
opening connection to acme-staging-v02.api.letsencrypt.org:443...
opened
starting SSL for acme-staging-v02.api.letsencrypt.org:443...
SSL established, protocol: TLSv1.3, cipher: TLS_AES_256_GCM_SHA384
opening connection to acme-staging-v02.api.letsencrypt.org:443...
opened
starting SSL for acme-staging-v02.api.letsencrypt.org:443...
SSL established, protocol: TLSv1.3, cipher: TLS_AES_256_GCM_SHA384
opening connection to acme-staging-v02.api.letsencrypt.org:443...
opened
starting SSL for acme-staging-v02.api.letsencrypt.org:443...
SSL established, protocol: TLSv1.3, cipher: TLS_AES_256_GCM_SHA384
opening connection to acme-staging-v02.api.letsencrypt.org:443...
opened
starting SSL for acme-staging-v02.api.letsencrypt.org:443...
SSL established, protocol: TLSv1.3, cipher: TLS_AES_256_GCM_SHA384
opening connection to acme-staging-v02.api.letsencrypt.org:443...
opened
starting SSL for acme-staging-v02.api.letsencrypt.org:443...
SSL established, protocol: TLSv1.3, cipher: TLS_AES_256_GCM_SHA384
opening connection to acme-staging-v02.api.letsencrypt.org:443...
opened
starting SSL for acme-staging-v02.api.letsencrypt.org:443...
SSL established, protocol: TLSv1.3, cipher: TLS_AES_256_GCM_SHA384
      * directory[/var/opt/gitlab/nginx/www/.well-known/acme-challenge] action nothing (skipped due to action :nothing)
      * directory[/var/opt/gitlab/nginx/www/.well-known/acme-challenge] action create (up to date)
      * file[/var/opt/gitlab/nginx/www/.well-known/acme-challenge/BGhKOmAq29bvABATtGouPgNiqBZvCfNN3gb9Fd2lYUg] action nothing (skipped due to action :nothing)
      * file[/var/opt/gitlab/nginx/www/.well-known/acme-challenge/BGhKOmAq29bvABATtGouPgNiqBZvCfNN3gb9Fd2lYUg] action create[2024-05-05T11:45:43+00:00] INFO: file[/var/opt/gitlab/nginx/www/.well-known/acme-challenge/BGhKOmAq29bvABATtGouPgNiqBZvCfNN3gb9Fd2lYUg] created file /var/opt/gitlab/nginx/www/.well-known/acme-challenge/BGhKOmAq29bvABATtGouPgNiqBZvCfNN3gb9Fd2lYUg

        - create new file /var/opt/gitlab/nginx/www/.well-known/acme-challenge/BGhKOmAq29bvABATtGouPgNiqBZvCfNN3gb9Fd2lYUg[2024-05-05T11:45:43+00:00] INFO: file[/var/opt/gitlab/nginx/www/.well-known/acme-challenge/BGhKOmAq29bvABATtGouPgNiqBZvCfNN3gb9Fd2lYUg] updated file contents /var/opt/gitlab/nginx/www/.well-known/acme-challenge/BGhKOmAq29bvABATtGouPgNiqBZvCfNN3gb9Fd2lYUg

        - update content in file /var/opt/gitlab/nginx/www/.well-known/acme-challenge/BGhKOmAq29bvABATtGouPgNiqBZvCfNN3gb9Fd2lYUg from none to 4cc057
        --- /var/opt/gitlab/nginx/www/.well-known/acme-challenge/BGhKOmAq29bvABATtGouPgNiqBZvCfNN3gb9Fd2lYUg    2024-05-05 11:45:43.476360079 +0000
        +++ /var/opt/gitlab/nginx/www/.well-known/acme-challenge/.chef-BGhKOmAq29bvABATtGouPgNiqBZvCfNN3gb9Fd2lYUg20240505-1030-auou5b  2024-05-05 11:45:43.476360079 +0000
        @@ -1 +1,2 @@
        +BGhKOmAq29bvABATtGouPgNiqBZvCfNN3gb9Fd2lYUg.KJAi4jIbPqQWEAQpCQburGmV0vU-XXM2KQqp4n6a-o8[2024-05-05T11:45:43+00:00] INFO: file[/var/opt/gitlab/nginx/www/.well-known/acme-challenge/BGhKOmAq29bvABATtGouPgNiqBZvCfNN3gb9Fd2lYUg] owner changed to 0
[2024-05-05T11:45:43+00:00] INFO: file[/var/opt/gitlab/nginx/www/.well-known/acme-challenge/BGhKOmAq29bvABATtGouPgNiqBZvCfNN3gb9Fd2lYUg] group changed to 0
[2024-05-05T11:45:43+00:00] INFO: file[/var/opt/gitlab/nginx/www/.well-known/acme-challenge/BGhKOmAq29bvABATtGouPgNiqBZvCfNN3gb9Fd2lYUg] mode changed to 644

        - change mode from '' to '0644'
        - change owner from '' to 'root'
        - change group from '' to 'root'
opening connection to acme-staging-v02.api.letsencrypt.org:443...
opened
starting SSL for acme-staging-v02.api.letsencrypt.org:443...
SSL established, protocol: TLSv1.3, cipher: TLS_AES_256_GCM_SHA384
opening connection to acme-staging-v02.api.letsencrypt.org:443...
opened
starting SSL for acme-staging-v02.api.letsencrypt.org:443...
SSL established, protocol: TLSv1.3, cipher: TLS_AES_256_GCM_SHA384
opening connection to acme-staging-v02.api.letsencrypt.org:443...
opened
starting SSL for acme-staging-v02.api.letsencrypt.org:443...
SSL established, protocol: TLSv1.3, cipher: TLS_AES_256_GCM_SHA384
      * file[/var/opt/gitlab/nginx/www/.well-known/acme-challenge/BGhKOmAq29bvABATtGouPgNiqBZvCfNN3gb9Fd2lYUg] action delete[2024-05-05T11:45:46+00:00] INFO: file[/var/opt/gitlab/nginx/www/.well-known/acme-challenge/BGhKOmAq29bvABATtGouPgNiqBZvCfNN3gb9Fd2lYUg] deleted file at /var/opt/gitlab/nginx/www/.well-known/acme-challenge/BGhKOmAq29bvABATtGouPgNiqBZvCfNN3gb9Fd2lYUg

        - delete file /var/opt/gitlab/nginx/www/.well-known/acme-challenge/BGhKOmAq29bvABATtGouPgNiqBZvCfNN3gb9Fd2lYUg
      * ruby_block[create certificate for gitlab.mydomain.de] action run

        ================================================================================
        Error executing action `run` on resource 'ruby_block[create certificate for gitlab.mydomain.de]'
        ================================================================================

        RuntimeError
        ------------
        [gitlab.mydomain.de] Validation failed, unable to request certificate, Errors: [{url: https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/12251661174/2AUnjA, status: invalid, error: {"type"=>"urn:ietf:params:acme:error:unauthorized", "detail"=>"myip: Invalid response from https://gitlab.mydomain.de/.well-known/acme-challenge/BGhKOmAq29bvABATtGouPgNiqBZvCfNN3gb9Fd2lYUg: 404", "status"=>403}} ]

        Cookbook Trace: (most recent call first)
        ----------------------------------------
        /opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/resources/certificate.rb:117:in `block (3 levels) in class_from_file'
        /opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/resources/certificate.rb:110:in `block in class_from_file'
        /opt/gitlab/embedded/cookbooks/cache/cookbooks/letsencrypt/resources/certificate.rb:43:in `block in class_from_file'

        Resource Declaration:
        ---------------------
        # In /opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/resources/certificate.rb

        110:     ruby_block "create certificate for #{new_resource.cn}" do
        111:       block do
        112:         unless (all_validations.map { |authz| authz.status == 'valid' }).all?
        113:           errors = all_validations.select { |authz| authz.status != 'valid' }.map do |authz|
        114:             "{url: #{authz.url}, status: #{authz.status}, error: #{authz.error}} "
        115:           end.reduce(:+)
        116:
        117:           raise "[#{new_resource.cn}] Validation failed, unable to request certificate, Errors: [#{errors}]"
        118:         end
        119:
        120:         begin
        121:           newcert = acme_cert(order, new_resource.cn, mykey, new_resource.alt_names)
        122:         rescue Acme::Client::Error => e
        123:           raise "[#{new_resource.cn}] Certificate request failed: #{e.message}"
        124:         else
        125:           Chef::Resource::File.new("#{new_resource.cn} SSL new crt", run_context).tap do |f|
        126:             f.path    new_resource.crt
        127:             f.owner   new_resource.owner
        128:             f.group   new_resource.group
        129:             f.content newcert
        130:             f.mode    00644
        131:           end.run_action :create
        132:         end
        133:       end
        134:     end
        135:   end

        Compiled Resource:
        ------------------
        # Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/resources/certificate.rb:110:in `block in class_from_file'

        ruby_block("create certificate for gitlab.mydomain.de") do
          action [:run]
          default_guard_interpreter :default
          declared_type :ruby_block
          cookbook_name "letsencrypt"
          recipe_name "http_authorization"
          block #<Proc:0x00007fbe423eee98 /opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/resources/certificate.rb:111>
        end

        System Info:
        ------------
        chef_version=18.3.0
        platform=ubuntu
        platform_version=22.04
        ruby=ruby 3.1.4p223 (2023-03-30 revision 957bb7cb81) [x86_64-linux]
        program_name=/opt/gitlab/embedded/bin/cinc-client
        executable=/opt/gitlab/embedded/bin/cinc-client


      ================================================================================
      Error executing action `create` on resource 'acme_certificate[staging]'
      ================================================================================

      RuntimeError
      ------------
      ruby_block[create certificate for gitlab.mydomain.de] (letsencrypt::http_authorization line 110) had an error: RuntimeError: [gitlab.mydomain.de] Validation failed, unable to request certificate, Errors: [{url: https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/12251661174/2AUnjA, status: invalid, error: {"type"=>"urn:ietf:params:acme:error:unauthorized", "detail"=>"myip: Invalid response from https://gitlab.mydomain.de/.well-known/acme-challenge/BGhKOmAq29bvABATtGouPgNiqBZvCfNN3gb9Fd2lYUg: 404", "status"=>403}} ]

      Cookbook Trace: (most recent call first)
      ----------------------------------------
      /opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/resources/certificate.rb:117:in `block (3 levels) in class_from_file'
      /opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/resources/certificate.rb:110:in `block in class_from_file'
      /opt/gitlab/embedded/cookbooks/cache/cookbooks/letsencrypt/resources/certificate.rb:43:in `block in class_from_file'

      Resource Declaration:
      ---------------------
      suppressed sensitive resource output

      Compiled Resource:
      ------------------
      suppressed sensitive resource output

      System Info:
      ------------
      chef_version=18.3.0
      platform=ubuntu
      platform_version=22.04
      ruby=ruby 3.1.4p223 (2023-03-30 revision 957bb7cb81) [x86_64-linux]
      program_name=/opt/gitlab/embedded/bin/cinc-client
      executable=/opt/gitlab/embedded/bin/cinc-client


    ================================================================================
    Error executing action `create` on resource 'letsencrypt_certificate[gitlab.mydomain.de]'
    ================================================================================

    RuntimeError
    ------------
    acme_certificate[staging] (letsencrypt::http_authorization line 43) had an error: RuntimeError: ruby_block[create certificate for gitlab.mydomain.de] (letsencrypt::http_authorization line 110) had an error: RuntimeError: [gitlab.mydomain.de] Validation failed, unable to request certificate, Errors: [{url: https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/12251661174/2AUnjA, status: invalid, error: {"type"=>"urn:ietf:params:acme:error:unauthorized", "detail"=>"myip: Invalid response from https://gitlab.mydomain.de/.well-known/acme-challenge/BGhKOmAq29bvABATtGouPgNiqBZvCfNN3gb9Fd2lYUg: 404", "status"=>403}} ]

    Cookbook Trace: (most recent call first)
    ----------------------------------------
    /opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/resources/certificate.rb:117:in `block (3 levels) in class_from_file'
    /opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/resources/certificate.rb:110:in `block in class_from_file'
    /opt/gitlab/embedded/cookbooks/cache/cookbooks/letsencrypt/resources/certificate.rb:43:in `block in class_from_file'

    Resource Declaration:
    ---------------------
    # In /opt/gitlab/embedded/cookbooks/cache/cookbooks/letsencrypt/recipes/http_authorization.rb

      6: letsencrypt_certificate site do
      7:   crt node['gitlab']['nginx']['ssl_certificate']
      8:   key node['gitlab']['nginx']['ssl_certificate_key']
      9:   notifies :run, "execute[reload nginx]", :immediate
     10:   notifies :run, 'ruby_block[display_le_message]'
     11:   only_if { omnibus_helper.service_up?('nginx') }
     12: end

    Compiled Resource:
    ------------------
    # Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/letsencrypt/recipes/http_authorization.rb:6:in `from_file'

    letsencrypt_certificate("gitlab.mydomain.de") do
      action [:create]
      updated true
      updated_by_last_action true
      default_guard_interpreter :default
      declared_type :letsencrypt_certificate
      cookbook_name "letsencrypt"
      recipe_name "http_authorization"
      crt "/etc/gitlab/ssl/gitlab.mydomain.de.crt"
      key "/etc/gitlab/ssl/gitlab.mydomain.de.key"
      cn "gitlab.mydomain.de"
      only_if { #code block }
    end

    System Info:
    ------------
    chef_version=18.3.0
    platform=ubuntu
    platform_version=22.04
    ruby=ruby 3.1.4p223 (2023-03-30 revision 957bb7cb81) [x86_64-linux]
    program_name=/opt/gitlab/embedded/bin/cinc-client
    executable=/opt/gitlab/embedded/bin/cinc-client

[2024-05-05T11:45:46+00:00] INFO: Running queued delayed notifications before re-raising exception

Running handlers:
[2024-05-05T11:45:46+00:00] ERROR: Running exception handlers
There was an error running gitlab-ctl reconfigure:

letsencrypt_certificate[gitlab.mydomain.de] (letsencrypt::http_authorization line 6) had an error: RuntimeError: acme_certificate[staging] (letsencrypt::http_authorization line 43) had an error: RuntimeError: ruby_block[create certificate for gitlab.mydomain.de] (letsencrypt::http_authorization line 110) had an error: RuntimeError: [gitlab.mydomain.de] Validation failed, unable to request certificate, Errors: [{url: https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/12251661174/2AUnjA, status: invalid, error: {"type"=>"urn:ietf:params:acme:error:unauthorized", "detail"=>"myip: Invalid response from https://gitlab.mydomain.de/.well-known/acme-challenge/BGhKOmAq29bvABATtGouPgNiqBZvCfNN3gb9Fd2lYUg: 404", "status"=>403}} ]

Running handlers complete
[2024-05-05T11:45:46+00:00] ERROR: Exception handlers complete
Infra Phase failed. 4 resources updated in 18 seconds
[2024-05-05T11:45:46+00:00] FATAL: Stacktrace dumped to /opt/gitlab/embedded/cookbooks/cache/cinc-stacktrace.out
[2024-05-05T11:45:46+00:00] FATAL: ---------------------------------------------------------------------------------------
[2024-05-05T11:45:46+00:00] FATAL: PLEASE PROVIDE THE CONTENTS OF THE stacktrace.out FILE (above) IF YOU FILE A BUG REPORT
[2024-05-05T11:45:46+00:00] FATAL: ---------------------------------------------------------------------------------------
[2024-05-05T11:45:46+00:00] FATAL: RuntimeError: letsencrypt_certificate[gitlab.mydomain.de] (letsencrypt::http_authorization line 6) had an error: RuntimeError: acme_certificate[staging] (letsencrypt::http_authorization line 43) had an error: RuntimeError: ruby_block[create certificate for gitlab.mydomain.de] (letsencrypt::http_authorization line 110) had an error: RuntimeError: [gitlab.mydomain.de] Validation failed, unable to request certificate, Errors: [{url: https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/12251661174/2AUnjA, status: invalid, error: {"type"=>"urn:ietf:params:acme:error:unauthorized", "detail"=>"myip: Invalid response from https://gitlab.mydomain.de/.well-known/acme-challenge/BGhKOmAq29bvABATtGouPgNiqBZvCfNN3gb9Fd2lYUg: 404", "status"=>403}} ]
Running reconfigure: NOT OK
== Reverting ==
== Fatal error ==
Something went wrong during final reconfiguration, please check the output
ok: down: postgresql: 1s, normally up
Symlink correct version of binaries: OK
ok: run: postgresql: (pid 1492) 0s
== Reverted ==
== Reverted to 13.14. Please check output for what went wrong ==
Toggling deploy page:rm -f /opt/gitlab/embedded/service/gitlab-rails/public/index.html
Toggling deploy page: OK
Toggling services:ok: run: alertmanager: (pid 1504) 1s
ok: run: crond: (pid 1513) 0s
ok: run: gitaly: (pid 1521) 0s
ok: run: gitlab-exporter: (pid 1540) 0s
ok: run: gitlab-kas: (pid 1543) 0s
ok: run: logrotate: (pid 1553) 1s
ok: run: postgres-exporter: (pid 1559) 0s
ok: run: prometheus: (pid 1571) 1s
ok: run: redis-exporter: (pid 1581) 0s
ok: run: registry: (pid 1583) 0s
ok: run: sidekiq: (pid 1589) 0s
ok: run: sshd: (pid 1594) 0s
Toggling services: OK
Upgrading the existing database failed and was reverted.
Please check the output, and open an issue at:
https://gitlab.com/gitlab-org/omnibus-gitlab/issues
If you would like to restart the instance without attempting to
upgrade, add the following to your docker command:
-e GITLAB_SKIP_PG_UPGRADE=true

After this output the container restarts and gives me the same logs again.

I have gitlab running behind a NGINX reverse proxy manager.

after the update did not work i tried to start gitlab with 16.11.0 again. unfortunately this did not work too. i got the same errors there.

The Docker-Compose:

version: '3.6'
services:
  web:
    image: 'gitlab/gitlab-ce:16.11.0-ce.0'
    restart: always
    hostname: 'gitlab.mydomain.de'
    environment:
      GITLAB_OMNIBUS_CONFIG: |
        gitlab_rails['initial_root_password'] = password'
        # Add any other gitlab.rb configuration here, each on its own line
    ports:
      - '8081:80'
      - '8082:443'
      - '23:22'
      - '587:587'
    volumes:
      - '$GITLAB_HOME/config:/etc/gitlab'
      - '$GITLAB_HOME/logs:/var/log/gitlab'
      - '$GITLAB_HOME/data:/var/opt/gitlab'
    shm_size: '256m'

I hope you can help me with my problem

Thank you

Here is another Logfile:

[2024-05-06T19:03:04+00:00] ERROR: Exception handlers complete
Infra Phase failed. 264 resources updated in 01 minutes 58 seconds
[2024-05-06T19:03:04+00:00] FATAL: Stacktrace dumped to /opt/gitlab/embedded/cookbooks/cache/cinc-stacktrace.out
[2024-05-06T19:03:04+00:00] FATAL: ---------------------------------------------------------------------------------------
[2024-05-06T19:03:04+00:00] FATAL: PLEASE PROVIDE THE CONTENTS OF THE stacktrace.out FILE (above) IF YOU FILE A BUG REPORT
[2024-05-06T19:03:04+00:00] FATAL: ---------------------------------------------------------------------------------------
[2024-05-06T19:03:04+00:00] FATAL: RuntimeError: letsencrypt_certificate[gitlab.mydomain.de] (letsencrypt::http_authorization line 6) had an error: RuntimeError: acme_certificate[staging] (letsencrypt::http_authorization line 43) had an error: RuntimeError: ruby_block[create certificate for gitlab.mydomain.de] (letsencrypt::http_authorization line 110) had an error: RuntimeError: [gitlab.mydomain.de] Validation failed, unable to request certificate, Errors: [{url: https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/12265126824/Kne2Cw, status: invalid, error: {"type"=>"urn:ietf:params:acme:error:unauthorized", "detail"=>"myip: Invalid response from https://gitlab.mydomain.de/.well-known/acme-challenge/DXCJRPunkTigUccjKUG2_11Qao9UTmGHuNWXJsf-E_k: 404", "status"=>403}} ]

and here is the output from cinc-stacktrace.out

root@gitlab:/# cat /opt/gitlab/embedded/cookbooks/cache/cinc-stacktrace.out
Generated at 2024-05-06 19:08:40 +0000
RuntimeError: letsencrypt_certificate[gitlab.mydomain.de] (letsencrypt::http_authorization line 6) had an error: RuntimeError: acme_certificate[staging] (letsencrypt::http_authorization line 43) had an error: RuntimeError: ruby_block[create certificate for gitlab.mydomain.de] (letsencrypt::http_authorization line 110) had an error: RuntimeError: [gitlab.mydomain.de] Validation failed, unable to request certificate, Errors: [{url: https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/12265174074/sToVMQ, status: invalid, error: {"type"=>"urn:ietf:params:acme:error:unauthorized", "detail"=>"myip: Invalid response from https://gitlab.mydomain.de/.well-known/acme-challenge/JT43XTc3PE47Y5hIW8ozxoDhhDmxSAGChLqzFEJ6UqY: 404", "status"=>403}} ]
/opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/resources/certificate.rb:117:in `block (3 levels) in class_from_file'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/provider/ruby_block.rb:31:in `block (2 levels) in <class:RubyBlock>'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/mixin/why_run.rb:51:in `add_action'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/provider.rb:293:in `converge_by'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/provider/ruby_block.rb:30:in `block in <class:RubyBlock>'
(eval):2:in `block in action_run'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/provider.rb:304:in `instance_eval'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/provider.rb:304:in `compile_and_converge_action'
(eval):2:in `action_run'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/provider.rb:245:in `run_action'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/resource.rb:601:in `block in run_action'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/resource.rb:628:in `with_umask'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/resource.rb:600:in `run_action'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/runner.rb:74:in `run_action'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/runner.rb:108:in `block in run_all_actions'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/runner.rb:108:in `each'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/runner.rb:108:in `run_all_actions'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/resource_collection.rb:64:in `insert'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/dsl/declare_resource.rb:267:in `declare_resource'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/dsl/resources.rb:36:in `ruby_block'
/opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/resources/certificate.rb:110:in `block in class_from_file'
(eval):2:in `block in action_create'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/provider.rb:304:in `instance_eval'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/provider.rb:304:in `compile_and_converge_action'
(eval):2:in `action_create'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/provider.rb:245:in `run_action'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/resource.rb:601:in `block in run_action'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/resource.rb:628:in `with_umask'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/resource.rb:600:in `run_action'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/runner.rb:74:in `run_action'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/runner.rb:108:in `block in run_all_actions'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/runner.rb:108:in `each'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/runner.rb:108:in `run_all_actions'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/resource_collection.rb:64:in `insert'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/dsl/declare_resource.rb:267:in `declare_resource'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/dsl/resources.rb:36:in `acme_certificate'
/opt/gitlab/embedded/cookbooks/cache/cookbooks/letsencrypt/resources/certificate.rb:43:in `block in class_from_file'
(eval):2:in `block in action_create'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/provider.rb:304:in `instance_eval'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/provider.rb:304:in `compile_and_converge_action'
(eval):2:in `action_create'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/provider.rb:245:in `run_action'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/resource.rb:601:in `block in run_action'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/resource.rb:628:in `with_umask'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/resource.rb:600:in `run_action'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/runner.rb:74:in `run_action'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/runner.rb:108:in `block in run_all_actions'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/runner.rb:108:in `each'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/runner.rb:108:in `run_all_actions'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/runner.rb:132:in `block in converge'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/resource_collection/resource_list.rb:96:in `block in execute_each_resource'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/resource_collection/stepable_iterator.rb:114:in `call_iterator_block'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/resource_collection/stepable_iterator.rb:85:in `step'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/resource_collection/stepable_iterator.rb:103:in `iterate'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/resource_collection/stepable_iterator.rb:54:in `each_with_index'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/resource_collection/resource_list.rb:94:in `execute_each_resource'
/opt/gitlab/embedded/lib/ruby/3.1.0/forwardable.rb:238:in `execute_each_resource'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/runner.rb:130:in `converge'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/client.rb:875:in `block in converge'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/client.rb:870:in `catch'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/client.rb:870:in `converge'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/client.rb:894:in `converge_and_save'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/client.rb:298:in `run'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/application.rb:305:in `run_with_graceful_exit_option'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/application.rb:281:in `block in run_chef_client'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/local_mode.rb:42:in `with_server_connectivity'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/application.rb:264:in `run_chef_client'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/application/base.rb:354:in `run_application'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/application.rb:67:in `run'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-bin-18.3.0/bin/cinc-client:25:in `<top (required)>'
/opt/gitlab/embedded/bin/cinc-client:25:in `load'
/opt/gitlab/embedded/bin/cinc-client:25:in `<main>'

>>>> Caused by RuntimeError: [gitlab.mydomain.de] Validation failed, unable to request certificate, Errors: [{url: https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/12265174074/sToVMQ, status: invalid, error: {"type"=>"urn:ietf:params:acme:error:unauthorized", "detail"=>"myip: Invalid response from https://gitlab.mydomain.de/.well-known/acme-challenge/JT43XTc3PE47Y5hIW8ozxoDhhDmxSAGChLqzFEJ6UqY: 404", "status"=>403}} ]
/opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/resources/certificate.rb:117:in `block (3 levels) in class_from_file'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/provider/ruby_block.rb:31:in `block (2 levels) in <class:RubyBlock>'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/mixin/why_run.rb:51:in `add_action'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/provider.rb:293:in `converge_by'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/provider/ruby_block.rb:30:in `block in <class:RubyBlock>'
(eval):2:in `block in action_run'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/provider.rb:304:in `instance_eval'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/provider.rb:304:in `compile_and_converge_action'
(eval):2:in `action_run'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/provider.rb:245:in `run_action'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/resource.rb:601:in `block in run_action'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/resource.rb:628:in `with_umask'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/resource.rb:600:in `run_action'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/runner.rb:74:in `run_action'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/runner.rb:108:in `block in run_all_actions'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/runner.rb:108:in `each'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/runner.rb:108:in `run_all_actions'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/resource_collection.rb:64:in `insert'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/dsl/declare_resource.rb:267:in `declare_resource'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/dsl/resources.rb:36:in `ruby_block'
/opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/resources/certificate.rb:110:in `block in class_from_file'
(eval):2:in `block in action_create'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/provider.rb:304:in `instance_eval'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/provider.rb:304:in `compile_and_converge_action'
(eval):2:in `action_create'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/provider.rb:245:in `run_action'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/resource.rb:601:in `block in run_action'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/resource.rb:628:in `with_umask'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/resource.rb:600:in `run_action'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/runner.rb:74:in `run_action'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/runner.rb:108:in `block in run_all_actions'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/runner.rb:108:in `each'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/runner.rb:108:in `run_all_actions'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/resource_collection.rb:64:in `insert'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/dsl/declare_resource.rb:267:in `declare_resource'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/dsl/resources.rb:36:in `acme_certificate'
/opt/gitlab/embedded/cookbooks/cache/cookbooks/letsencrypt/resources/certificate.rb:43:in `block in class_from_file'
(eval):2:in `block in action_create'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/provider.rb:304:in `instance_eval'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/provider.rb:304:in `compile_and_converge_action'
(eval):2:in `action_create'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/provider.rb:245:in `run_action'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/resource.rb:601:in `block in run_action'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/resource.rb:628:in `with_umask'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/resource.rb:600:in `run_action'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/runner.rb:74:in `run_action'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/runner.rb:108:in `block in run_all_actions'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/runner.rb:108:in `each'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/runner.rb:108:in `run_all_actions'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/runner.rb:132:in `block in converge'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/resource_collection/resource_list.rb:96:in `block in execute_each_resource'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/resource_collection/stepable_iterator.rb:114:in `call_iterator_block'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/resource_collection/stepable_iterator.rb:85:in `step'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/resource_collection/stepable_iterator.rb:103:in `iterate'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/resource_collection/stepable_iterator.rb:54:in `each_with_index'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/resource_collection/resource_list.rb:94:in `execute_each_resource'
/opt/gitlab/embedded/lib/ruby/3.1.0/forwardable.rb:238:in `execute_each_resource'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/runner.rb:130:in `converge'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/client.rb:875:in `block in converge'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/client.rb:870:in `catch'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/client.rb:870:in `converge'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/client.rb:894:in `converge_and_save'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/client.rb:298:in `run'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/application.rb:305:in `run_with_graceful_exit_option'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/application.rb:281:in `block in run_chef_client'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/local_mode.rb:42:in `with_server_connectivity'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/application.rb:264:in `run_chef_client'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/application/base.rb:354:in `run_application'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/application.rb:67:in `run'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-bin-18.3.0/bin/cinc-client:25:in `<top (required)>'
/opt/gitlab/embedded/bin/cinc-client:25:in `load'
/opt/gitlab/embedded/bin/cinc-client:25:in `<main>'root@gitlab:/#

Thank you for your help