GitLab update fails saying needs to run reconfigure which also fails

System is ubuntu 20.04.3. It’s a fresh install on new hardware, installed GitLab 14.5.2 and restored a backup from the previous server. I copied the old secrets files, but I had to manualy reset users passwords after the restore.

Gitlab is working fine behind the servers own Nginx as a reverse proxy, a setup I have used for years, so as a result Gitlab is configured not to start it’s own nginx process in gitlab.rb → nginx[‘enable’] = false

But I can’t update it. It seems to fail trying to enable nginx which is disabled in gitlab.rb. The file /opt/gitlab/sv/nginx/log/config does exist

Here is the tail end of the reconfigure output

Recipe: letsencrypt::enable
  * ruby_block[http external-url] action run (skipped due to only_if)
Recipe: nginx::enable
  * runit_service[nginx] action enable
    * ruby_block[restart_service] action nothing (skipped due to action :nothing)
    * ruby_block[restart_log_service] action nothing (skipped due to action :nothing)
    * ruby_block[reload_log_service] action nothing (skipped due to action :nothing)
    * directory[/opt/gitlab/sv/nginx] action create (up to date)
    * template[/opt/gitlab/sv/nginx/run] action create (up to date)
    * directory[/opt/gitlab/sv/nginx/log] action create (up to date)
    * directory[/opt/gitlab/sv/nginx/log/main] action create (up to date)
    * template[/opt/gitlab/sv/nginx/log/config] action create (up to date)
    * ruby_block[verify_chown_persisted_on_nginx] action nothing (skipped due to action :nothing)
    * link[/var/log/gitlab/nginx/config] action create

      ================================================================================
      Error executing action `create` on resource 'link[/var/log/gitlab/nginx/config]'
      ================================================================================

      Errno::ENOENT
      -------------
      No such file or directory @ rb_file_s_symlink - (/opt/gitlab/sv/nginx/log/config, /var/log/gitlab/nginx/config)

      Cookbook Trace:
      ---------------
      /opt/gitlab/embedded/cookbooks/cache/cookbooks/runit/libraries/provider_runit_service.rb:295:in `block in <class:RunitService>'

      Resource Declaration:
      ---------------------
      # In /opt/gitlab/embedded/cookbooks/cache/cookbooks/runit/libraries/provider_runit_service.rb

      144:             link ::File.join(new_resource.options[:log_directory], 'config') do
      145:               to ::File.join(sv_dir_name, 'log', 'config')
      146:             end
      147:

      Compiled Resource:
      ------------------
      # Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/runit/libraries/provider_runit_service.rb:144:in `block in <class:RunitService>'

      link("/var/log/gitlab/nginx/config") do
        action [:create]
        default_guard_interpreter :default
        declared_type :link
        cookbook_name "nginx"
        to "/opt/gitlab/sv/nginx/log/config"
        target_file "/var/log/gitlab/nginx/config"
      end

      System Info:
      ------------
      chef_version=15.17.4
      platform=ubuntu
      platform_version=20.04
      ruby=ruby 2.7.5p203 (2021-11-24 revision f69aeb8314) [x86_64-linux]
      program_name=/opt/gitlab/embedded/bin/chef-client
      executable=/opt/gitlab/embedded/bin/chef-client


    ================================================================================
    Error executing action `enable` on resource 'runit_service[nginx]'
    ================================================================================

    Errno::ENOENT
    -------------
    link[/var/log/gitlab/nginx/config] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/runit/libraries/provider_runit_service.rb line 144) had an error: Errno::ENOENT: No such file or directory @ rb_file_s_symlink - (/opt/gitlab/sv/nginx/log/config, /var/log/gitlab/nginx/config)

    Cookbook Trace:
    ---------------
    /opt/gitlab/embedded/cookbooks/cache/cookbooks/runit/libraries/provider_runit_service.rb:295:in `block in <class:RunitService>'

    Resource Declaration:
    ---------------------
    # In /opt/gitlab/embedded/cookbooks/cache/cookbooks/nginx/recipes/enable.rb

     18: runit_service "nginx" do
     19:   start_down node['gitlab']['nginx']['ha']
     20:   options({
     21:     log_directory: nginx_log_dir
     22:   }.merge(params))
     23:   log_options node['gitlab']['logging'].to_hash.merge(node['gitlab']['nginx'].to_hash)
     24: end
     25:

    Compiled Resource:
    ------------------
    # Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/nginx/recipes/enable.rb:18:in `from_file'

    runit_service("nginx") do
      action [:enable]
      default_guard_interpreter :default
      declared_type :runit_service
      cookbook_name "nginx"
      recipe_name "enable"
      start_down false
      service_name "nginx"
      options {:env_dir=>"/opt/gitlab/sv/nginx/env", :log_directory=>"/var/log/gitlab/nginx"}
      log_options {"svlogd_size"=>209715200, "svlogd_num"=>30, "svlogd_timeout"=>86400, "svlogd_filter"=>"gzip", "svlogd_udp"=>nil, "svlogd_prefix"=>nil, "udp_log_shipping_host"=>nil, "udp_log_shipping_hostname"=>nil, "udp_log_shipping_port"=>514, "logrotate_frequency"=>"daily", "logrotate_maxsize"=>nil, "logrotate_size"=>nil, "logrotate_rotate"=>30, "logrotate_compress"=>"compress", "logrotate_method"=>"copytruncate", "logrotate_postrotate"=>nil, "logrotate_dateformat"=>nil, "enable"=>false, "ha"=>false, "dir"=>"/var/opt/gitlab/nginx", "log_directory"=>"/var/log/gitlab/nginx", "error_log_level"=>"error", "worker_processes"=>32, "worker_connections"=>10240, "log_format"=>"$remote_addr - $remote_user [$time_local] \"$request_method $filtered_request_uri $server_protocol\" $status $body_bytes_sent \"$filtered_http_referer\" \"$http_user_agent\" $gzip_ratio", "sendfile"=>"on", "tcp_nopush"=>"on", "tcp_nodelay"=>"on", "hide_server_tokens"=>"off", "gzip_http_version"=>"1.1", "gzip_comp_level"=>"2", "gzip_proxied"=>"no-cache no-store private expired auth", "gzip_types"=>["text/plain", "text/css", "application/x-javascript", "text/xml", "application/xml", "application/xml+rss", "text/javascript", "application/json"], "keepalive_timeout"=>65, "keepalive_time"=>"1h", "client_max_body_size"=>"500m", "cache_max_size"=>"5000m", "redirect_http_to_https"=>true, "redirect_http_to_https_port"=>80, "request_buffering_off_path_regex"=>"/api/v\\d/jobs/\\d+/artifacts$|\\.git/git-receive-pack$|\\.git/gitlab-lfs/objects|\\.git/info/lfs/objects/batch$", "ssl_client_certificate"=>nil, "ssl_verify_client"=>nil, "ssl_verify_depth"=>"1", "ssl_certificate"=>"/etc/gitlab/ssl/git.nibiru.nz.crt", "ssl_certificate_key"=>"/etc/gitlab/ssl/git.nibiru.nz.key", "ssl_ciphers"=>"ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:AES256-GCM-SHA384", "ssl_prefer_server_ciphers"=>"off", "ssl_protocols"=>"TLSv1.2 TLSv1.3", "ssl_session_cache"=>"shared:SSL:10m", "ssl_session_tickets"=>"off", "ssl_session_timeout"=>"1d", "ssl_dhparam"=>nil, "listen_addresses"=>["*"], "listen_port"=>80, "listen_https"=>false, "custom_gitlab_server_config"=>nil, "custom_nginx_config"=>nil, "proxy_read_timeout"=>3600, "proxy_connect_timeout"=>300, "proxy_set_headers"=>{"Host"=>"$http_host_with_default", "X-Real-IP"=>"$remote_addr", "X-Forwarded-For"=>"$proxy_add_x_forwarded_for", "Upgrade"=>"$http_upgrade", "Connection"=>"$connection_upgrade", "X-Forwarded-Proto"=>"https", "X-Forwarded-Ssl"=>"on"}, "referrer_policy"=>"strict-origin-when-cross-origin", "http2_enabled"=>true, "proxy_cache_path"=>"proxy_cache keys_zone=gitlab:10m max_size=1g levels=1:2", "proxy_cache"=>"gitlab", "real_ip_trusted_addresses"=>[], "real_ip_header"=>nil, "real_ip_recursive"=>nil, "server_names_hash_bucket_size"=>64, "hsts_max_age"=>63072000, "hsts_include_subdomains"=>false, "gzip_enabled"=>true, "consul_service_name"=>"nginx", "consul_service_meta"=>nil, "status"=>{"enable"=>true, "listen_addresses"=>["*"], "fqdn"=>"localhost", "port"=>8060, "vts_enable"=>true, "options"=>{"server_tokens"=>"off", "access_log"=>"off", "allow"=>"127.0.0.1", "deny"=>"all"}}}
      run_template_name "nginx"
      log_template_name "nginx"
      check_script_template_name "nginx"
      finish_script_template_name "nginx"
      control_template_names {}
    end

    System Info:
    ------------
    chef_version=15.17.4
    platform=ubuntu
    platform_version=20.04
    ruby=ruby 2.7.5p203 (2021-11-24 revision f69aeb8314) [x86_64-linux]
    program_name=/opt/gitlab/embedded/bin/chef-client
    executable=/opt/gitlab/embedded/bin/chef-client


Running handlers:
There was an error running gitlab-ctl reconfigure:

runit_service[nginx] (nginx::enable line 18) had an error: Errno::ENOENT: link[/var/log/gitlab/nginx/config] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/runit/libraries/provider_runit_service.rb line 144) had an error: Errno::ENOENT: No such file or directory @ rb_file_s_symlink - (/opt/gitlab/sv/nginx/log/config, /var/log/gitlab/nginx/config)

Running handlers complete
Chef Infra Client failed. 0 resources updated in 12 seconds

Hi,

Seems to be a problem with /var/log/gitlab/nginx/config as it mentions a problem with no such file or directory @ rb_file_s_symlink.

Please verify that:

  1. /opt/gitlab/sv/nginx/log/config - this should exist and it is a file (which you mentioned in your post as existing).
  2. /var/log/gitlab/nginx/config - this should exist as a symlink.

This is how mine looks:

lrwxrwxrwx  1 root root         31 Sep 11  2020 config -> /opt/gitlab/sv/nginx/log/config

If config doesn’t exist under /var/opt/gitlab/nginx, then do this:

cd /var/log/gitlab/nginx
ln -s /opt/gitlab/sv/nginx/log/config

and then run your reconfigure again.

Hi, are those paths correct? /var/opt/gitlab/nginx/ does not exist on my system at all.

I created the link in both /var/opt/gitlab/ngnix and /var/log/gitlab/nginx and the result was the same error, no such file or directory

The contents of the config file are:
s209715200
n30
t86400
!gzip

Sorry, my bad:

/var/log/gitlab/nginx/config

that is the one that doesn’t exist. I should have written to create the symlink in that directory, will amend my post above.

ok, I did both just in case, same error, so even with a link it says it can find it, so just now I copied the file and it cant find it. Maybe it is file permissions?

No, I am an idiot and spelt nginx as ngnix :laughing:

now I have a new error so I’ll close this and start a new topic

1 Like

If you copied the file, I recommend replacing it with a symlink. This will be because the file will be updated during an upgrade of Gitlab and replaced with a new version. Therefore the one you copied will not show the updated info, unless it is a symlink. My file has the same content as yours currently since we are running the same version of Gitlab.

Yes, I did that

1 Like

And that seems to have solved my other issue, I stopped GitLab then restarted it and reconfigure now works and GitLab is currently updating to 14.6

Thanks for taking the time to answer :smiley:

My pleasure glad it’s working :slight_smile: