Omnibus gitlab 12 upgrade failed with permission denied gitlab.yml

I upgraded the 11.11.3 omnibus package to 11.11.8 successfully (and all of the upgrades in the past) , but couldn’t upgrade to 12.3.4:

Error executing action run on resource ‘bash[migrate gitlab-rails database]’
================================================================================

Mixlib::ShellOut::ShellCommandFailed
------------------------------------
Expected process to exit with [0], but received '1'
---- Begin output of "bash"  "/tmp/chef-script20191009-26110-146kujo" ----
STDOUT: rake aborted!
Errno::EACCES: Permission denied @ rb_sysopen - /opt/gitlab/embedded/service/gitlab-rails/config/gitlab.yml
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/1_settings.rb:6:in `<top (required)>'
/opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:6:in `<top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => gitlab:db:configure => environment
(See full trace by running task with --trace)
STDERR: 
---- End output of "bash"  "/tmp/chef-script20191009-26110-146kujo" ----


There was an error running gitlab-ctl reconfigure:

bash[migrate gitlab-rails database] (gitlab::database_migrations line 54) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received ‘1’
---- Begin output of “bash” “/tmp/chef-script20191009-26110-146kujo” ----
STDOUT: rake aborted!
Errno::EACCES: Permission denied @ rb_sysopen - /opt/gitlab/embedded/service/gitlab-rails/config/gitlab.yml
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/1_settings.rb:6:in <top (required)>' /opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:6:in<top (required)>’
/opt/gitlab/embedded/bin/bundle:23:in load' /opt/gitlab/embedded/bin/bundle:23:in
Tasks: TOP => gitlab:db:configure => environment
(See full trace by running task with --trace)
STDERR:
---- End output of “bash” “/tmp/chef-script20191009-26110-146kujo” ----
Ran “bash” “/tmp/chef-script20191009-26110-146kujo” returned 1

ls -la /opt/gitlab/embedded/service/gitlab-rails/config/gitlab.yml
lrwxrwxrwx. 1 root root 43 Jul 19 2018 /opt/gitlab/embedded/service/gitlab-rails/config/gitlab.yml -> /var/opt/gitlab/gitlab-rails/etc/gitlab.yml

sudo ls -la /var/opt/gitlab/gitlab-rails/etc/gitlab.yml
-rw-r-----+ 1 root git 21391 Oct 7 09:46 /var/opt/gitlab/gitlab-rails/etc/gitlab.yml

My user is a sudoer. With root same error.

Are you using an OS with selinux such as CentOS or Redhat? “ls -Z” might give you some more info then. Not sure what properties it should have, but selinux often blocks access (which it is supposed to do).

Thanks bartj!

sudo ls -Z /opt/gitlab/embedded/service/gitlab-rails/config/gitlab.yml
lrwxrwxrwx. root root unconfined_u:object_r:usr_t:s0 /opt/gitlab/embedded/service/gitlab-rails/config/gitlab.yml -> /var/opt/gitlab/gitlab-rails/etc/gitlab.yml

sudo ls -Z /var/opt/gitlab/gitlab-rails/etc/gitlab.ym
-rw-r-----+ root git ? /var/opt/gitlab/gitlab-rails/etc/gitlab.yml

CentOs 7, but selinux is disabled.
/var/log/gitlab$ sestatus
SELinux status: disabled

If it is disabled, it shouldn’t prevent access to files. Sorry, this was my best guess.