I use GitLab on version 16.10.1 and when trying to restore my backup (from the same version) I got the following error:
root@37f36c134253:/# gitlab-backup restore BACKUP=/var/opt/gitlab/backups/2024-04-12-16.10.1-ee
Rails couldn't infer whether you are using multiple databases from your database.yml and can't generate the tasks for the non-primary databases. If you'd like to use this feature, please simplify your ERB.
Rails couldn't infer whether you are using multiple databases from your database.yml and can't generate the tasks for the non-primary databases. If you'd like to use this feature, please simplify your ERB.
Rails couldn't infer whether you are using multiple databases from your database.yml and can't generate the tasks for the non-primary databases. If you'd like to use this feature, please simplify your ERB.
Rails couldn't infer whether you are using multiple databases from your database.yml and can't generate the tasks for the non-primary databases. If you'd like to use this feature, please simplify your ERB.
Rails couldn't infer whether you are using multiple databases from your database.yml and can't generate the tasks for the non-primary databases. If you'd like to use this feature, please simplify your ERB.
Rails couldn't infer whether you are using multiple databases from your database.yml and can't generate the tasks for the non-primary databases. If you'd like to use this feature, please simplify your ERB.
Rails Error: Unable to access log file. Please ensure that /opt/gitlab/embedded/service/gitlab-rails/log/production.log exists and is writable (i.e. make it writable for user and group: chmod 0664 /opt/gitlab/embedded/service/gitlab-rails/log/production.log). The log level has been raised to WARN and the output directed to STDERR until the problem is fixed.
rake aborted!
Cannot load database configuration:
File "/opt/gitlab/embedded/service/gitlab-rails/config/database.yml" is a symlink that does not point to a valid file
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/patch/database_config.rb:15:in `database_configuration'
/opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:7:in `<top (required)>'
<internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:37:in `require'
<internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:37:in `require'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/helpers.rake:7:in `block in <top (required)>'
/opt/gitlab/embedded/bin/bundle:25:in `load'
/opt/gitlab/embedded/bin/bundle:25:in `<main>'
Caused by:
File "/opt/gitlab/embedded/service/gitlab-rails/config/database.yml" is a symlink that does not point to a valid file
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/patch/database_config.rb:15:in `database_configuration'
/opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:7:in `<top (required)>'
<internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:37:in `require'
<internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:37:in `require'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/helpers.rake:7:in `block in <top (required)>'
/opt/gitlab/embedded/bin/bundle:25:in `load'
/opt/gitlab/embedded/bin/bundle:25:in `<main>'
Tasks: TOP => environment
(See full trace by running task with --trace)
root@37f36c134253:/#
I searched and didn’t find any related topics, does anyone have any idea what could be causing this error?
Looking at the error message, these lines stand out to me as indicator of root cause of your problem here:
Cannot load database configuration:
File "/opt/gitlab/embedded/service/gitlab-rails/config/database.yml" is a symlink that does not point to a valid file
Caused by:
File "/opt/gitlab/embedded/service/gitlab-rails/config/database.yml" is a symlink that does not point to a valid file
It sounds like your GitLab installation has a bad symlink or that /var/opt/gitlab/gitlab-rails/etc/database.yml was corrupted or deleted. Both the symlink and the file it points to should be created or updated when you make changes to gitlab.rb and run gitlab-ctl reconfigure.
I also see an unrelated issue with your log files:
Rails Error: Unable to access log file. Please ensure that /opt/gitlab/embedded/service/gitlab-rails/log/production.log exists and is writable (i.e. make it writable for user and group: chmod 0664 /opt/gitlab/embedded/service/gitlab-rails/log/production.log). The log level has been raised to WARN and the output directed to STDERR until the problem is fixed.
To troubleshoot further, can you please provide the output of the following commands?
ls -l /opt/gitlab/embedded/service/gitlab-rails/config/database.yml
ls -l /var/opt/gitlab/gitlab-rails/etc/database.yml
ls -l /opt/gitlab/embedded/service/gitlab-rails/log/production.log
ls -l /var/log/gitlab/gitlab-rails/production.log
Thanks for following up and sharing this information. Are you still encountering this problem?
The user/group ownership is what I’d expect to see in a default installation, however the . at the end of file mode/permissions indicates that there are SELinux security contexts associated with these files. I suspect your SELinux security context settings here could potentially be causing the error you’re seeing. If SELinux is enforcing policies that restrict access to the symlink or the target file, it could result in the “file is a symlink that does not point to a valid file” error.
The sestatus command will tell you if SELinux is currently enabled and enforcing policies. If SELinux is enforcing, it will indicate “Enforcing mode”. If it is in Enforcing mode, you might try checking the SELinux contexts of problematic files with ls -Z
For example
ls -Z /opt/gitlab/embedded/service/gitlab-rails/config/database.yml
ls -Z /var/opt/gitlab/gitlab-rails/etc/database.yml
If the root of the problem is SELinux context and policy enforcement, I suggest talking with your team about options to adjust SELinux contexts on these files or temporarily disabling SE Linux enforcement before/during backup restoration to unblock your ability to restore from backup.
Were you previously able to successfully restore GitLab backups and this is the first time you’re having problems, or is it possible this issue has been ongoing?
What you can also do after what @gitlab-greg suggested, is temporarily put the server in permissive mode if selinux is running, so:
setenforce 0
then attempt the restore and see if it’s any better. If it is, you can then use the selinux debugging tools to check what avc alerts were made during permissive mode. These can then help you fix it. To find out the avc violations, do:
audit2allow -a /var/log/audit/audit.log
this is on the assumption that it is running an EL-based distro like RHEL. Amend the path where necessary to the log file if a different distro. Once you’ve fixed the violations, you can then re-enable selinux by doing:
setenforce 1
you will now be back in Enforcing mode instead of permissive now. This means that subsequent actions will be blocked unless the avc violations were addressed.
One more key thing, before a restore, you do have to do a:
gitlab-ctl reconfigure
to ensure that at least a basic empty installation of Gitlab is running. This ensures all services are running before you start the restore. I’m assuming you did do this already, but just in case. On a new server prior to restore, it’s imperative to do:
Install Gitlab package.
Place /etc/gitlab/gitlab.rb and /etc/gitlab/gitlab-secrets.json on the new server.
Run gitlab-ctl-reconfigure to get a basic empty system.
