Error "Cannot load database configuration" : symlink does not point to a valid file

Using a self-maanged gitlab-ee in version 12.10.3 on ubuntu (Bionic), I cannot start (spawn) the process to get the website in a browser. It was previously running for about 6 months, but since two weeks, it is down.

I upgraded today to to version 12.10.3 to try to solve the issue. upgrade successful, but nothing changed.

This is the error shown on the browser (not a phusion passenger error) :

The Phusion Passenger application server tried to start the web application. But the application itself (and not Passenger) encountered an internal error.

Error details:

Cannot load database configuration:
File "/opt/gitlab/embedded/service/gitlab-rails/config/database.yml" is a symlink that does not point to a valid file (RuntimeError)

In console, the file seems to exists and accessible, but the symbolik link is written in red in the console :

ls -als /opt/gitlab/embedded/service/gitlab-rails/config/database.yml
0 lrwxrwxrwx 1 root root 45 May 11 09:07 /opt/gitlab/embedded/service/gitlab-rails/config/database.yml -> /var/opt/gitlab/gitlab-rails/etc/database.yml


sudo cat /opt/gitlab/embedded/service/gitlab-rails/config/database.yml
# This file is managed by gitlab-ctl. Manual changes will be
# erased! To change the contents below, edit /etc/gitlab/gitlab.rb
# and run `sudo gitlab-ctl reconfigure`.
production:
  adapter: postgresql
  encoding: unicode
  collation: 
  database: gitlabhq_production
  pool: 1
  username: "gitlab"
  password: 
  host: "/var/opt/gitlab/postgresql"
  port: 5432
  socket: 
  sslmode: 
  sslcompression: 0
  sslrootcert: 
  sslca: 
  load_balancing: {"hosts":[]}
  prepared_statements: false
  statement_limit: 1000
  fdw: 
  variables:
    statement_timeout: 60000

I suppose there is an issue with users and groups, but how can I figure that… User “gitlab” does not exist (but it worked in the past, so why should this have changed ?)

groups gitlab
groups: ‘gitlab’: no such user

Output of gitlab-ctl seems to be OK.

sudo gitlab-ctl status
down: alertmanager: 0s, normally up, want up; run: log: (pid 2878) 3450857s
run: crond: (pid 12557) 821s; run: log: (pid 2891) 3450856s
run: gitaly: (pid 12566) 820s; run: log: (pid 2880) 3450857s
run: gitlab-exporter: (pid 12572) 820s; run: log: (pid 2883) 3450856s
run: gitlab-workhorse: (pid 12581) 819s; run: log: (pid 2876) 3450857s
run: grafana: (pid 12598) 819s; run: log: (pid 2881) 3450856s
run: logrotate: (pid 12607) 818s; run: log: (pid 2882) 3450856s
run: node-exporter: (pid 12622) 818s; run: log: (pid 2888) 3450856s
run: postgres-exporter: (pid 12628) 818s; run: log: (pid 2889) 3450856s
run: postgresql: (pid 12646) 817s; run: log: (pid 2884) 3450856s
run: prometheus: (pid 12650) 817s; run: log: (pid 2886) 3450856s
run: redis: (pid 12665) 816s; run: log: (pid 2877) 3450857s
run: redis-exporter: (pid 12672) 816s; run: log: (pid 2885) 3450856s
run: sidekiq: (pid 12710) 813s; run: log: (pid 2875) 3450857s
run: unicorn: (pid 12736) 810s; run: log: (pid 2887) 3450856s

Thank you for helping !

Hi,

seems the symlink target /var/opt/gitlab/gitlab-rails/etc/database.yml got a problem. Please share a full analysis like

file /var/opt/gitlab/gitlab-rails/etc/database.yml

ls -lah /var/opt/gitlab/gitlab-rails/etc/database.yml

The symlink permissions don’t come into play, the target file’s permissions are important.

Cheers,
Michael

Thanks for helping !
I’ll try to provide all the necessary information :slight_smile:

root@saphir:/opt/gitlab/embedded/service/gitlab-rails/config# 
# file database.yml
database.yml: symbolic link to /var/opt/gitlab/gitlab-rails/etc/database.yml 

# ls -lah /var/opt/gitlab/gitlab-rails/etc/database.yml
-rw-r----- 1 root git 567 May 11 09:52 /var/opt/gitlab/gitlab-rails/etc/database.yml

# file /var/opt/gitlab/gitlab-rails/etc/database.yml
/var/opt/gitlab/gitlab-rails/etc/database.yml: ASCII text

It seems the file exists and is not corrupted.

Is there a issue with the username executing gitlab ? Maybe a confusion between “git” and “gitlab”

 cat gitlab.xxx.com
 server {
  server_name gitlab.xxx.com;
  server_tokens off;
  root /opt/gitlab/embedded/service/gitlab-rails/public;

  client_max_body_size 250m;
  access_log  /var/log/gitlab/nginx/gitlab_access.log;
  error_log   /var/log/gitlab/nginx/gitlab_error.log;

  passenger_friendly_error_pages on;

  # Ensure Passenger uses the bundled Ruby version
  passenger_ruby /opt/gitlab/embedded/bin/ruby;

  # Correct the $PATH variable to included packaged executables
  passenger_env_var PATH "/opt/gitlab/bin:/opt/gitlab/embedded/bin:/usr/local/bin:/usr/bin:/bin";

  # Make sure Passenger runs as the correct user and group to
  # prevent permission issues
  passenger_user gitlab;
  passenger_group gitlab;

  # Enable Passenger & keep at least one instance running at all times
  passenger_enabled on;
  passenger_min_instances 1;
  passenger_start_timeout 300;

  error_page 502 /502.html;

 [SSL stuff ...]

}
server {
    if ($host = gitlab.xxx.com) {
        return 301 https://$host$request_uri;
    } # managed by Certbot


  listen *:80;
  server_name gitlab.xxx.com;
    return 404; # managed by Certbot

}

Users in /etc/passwd :

gitlab-www
git
gitlab-redis
gitlab-psql
registry
gitlab-prometheus

groups gitlab
groups: ‘gitlab’: no such user

Can I test the database connection in a way ?
Or is there a way I can restore (reinstall) gitlab to restore missing link / file ?

This is the full error log :

The stdout/stderr output of the subprocess so far is:

 [passenger_native_support.so] trying to compile for the current user (nobody) and Ruby interpreter...
     (set PASSENGER_COMPILE_NATIVE_SUPPORT_BINARY=0 to disable)
     Warning: compilation didn't succeed. To learn why, read this file:
     /tmp/passenger_native_support-1jislu3.log
 [passenger_native_support.so] finding downloads for the current Ruby interpreter...
     (set PASSENGER_DOWNLOAD_NATIVE_SUPPORT_BINARY=0 to disable)
     Could not download https://github.com/phusion/passenger/releases/download/release-6.0.4/rubyext-ruby-2.6.5-x86_64-linux.tar.gz: The requested URL returned error: 404 Not Found
     Trying next mirror...
     Could not download https://oss-binaries.phusionpassenger.com/binaries/passenger/by_release/6.0.4/rubyext-ruby-2.6.5-x86_64-linux.tar.gz: The requested URL returned error: 404 Not Found
     Trying next mirror...
     Could not download https://s3.amazonaws.com/phusion-passenger/binaries/passenger/by_release/6.0.4/rubyext-ruby-2.6.5-x86_64-linux.tar.gz: The requested URL returned error: 403 Forbidden
 [passenger_native_support.so] will not be used (can't compile or download) 
  --> Passenger will still operate normally.
Rails Error: Unable to access log file. Please ensure that /opt/gitlab/embedded/service/gitlab-rails/log/production.log exists and is writable (ie, make it writable for user and group: chmod 0664 /opt/gitlab/embedded/service/gitlab-rails/log/production.log). The log level has been raised to WARN and the output directed to STDERR until the problem is fixed.
Error: The application encountered the following error: Cannot load database configuration:
File "/opt/gitlab/embedded/service/gitlab-rails/config/database.yml" is a symlink that does not point to a valid file (RuntimeError)
    /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-6.0.2/lib/rails/paths.rb:213:in `block in existent'
    /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-6.0.2/lib/rails/paths.rb:209:in `select'
    /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-6.0.2/lib/rails/paths.rb:209:in `existent'