GitLab Omnibus changing permissions of `gitlab_workhorse_secret`

When performing an upgrade and therefore running reconfigure, and on all future runs of gitlab-ctl reconfigure the permissions of the .gitlab-workhorse-secret are changed to root:root, which means that gitlab-rails cannot open the file in order to read them and use them. This causes it to fail to migrate, during an upgrade, or fail to boot during normal operation. I can see from the chef recipe that it is designed to own it to root in every case, which is what I worked around in order to get us up and running.

I believe I caused the problem by specifying

gitlab_workhorse['secret_token'] = 'redacted'

in our gitlab.rb.erb, but i’m not sure how.

I’m fairly sure this is self-inflicted pain, but it’s unclear to me how.

Edit: The exception is lost in my terminal scrollback, but it was an ENOACCESS in the secret reader that has a rescue write_secret block. Both branches of that code gave ENOACCESS, which makes sense.

Edit: An wild exception appears:

This occurred immediately after installing GitLab 11.3.1 via the gitlab-ci dpkg. Note this time, the upgrade worked. I assume because there were no migrations and no database hooks involved, therefore the app didn’t need to be loaded during the gitlab-ctl reconfigure post install dpkg hook.

==> /var/log/gitlab/unicorn/unicorn_stderr.log <==
Errno::EACCES: Permission denied @ rb_sysopen - /opt/gitlab/embedded/service/gitlab-rails/.gitlab_workhorse_secret
  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/workhorse.rb:180:in `initialize'
  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/workhorse.rb:180:in `open'
  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/workhorse.rb:180:in `write_secret'
  /opt/gitlab/embedded/service/gitlab-rails/config/initializers/gitlab_workhorse_secret.rb:4:in `rescue in <top (required)>'
  /opt/gitlab/embedded/service/gitlab-rails/config/initializers/gitlab_workhorse_secret.rb:1:in `<top (required)>'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activesupport-4.2.10/lib/active_support/dependencies.rb:268:in `load'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activesupport-4.2.10/lib/active_support/dependencies.rb:268:in `block in load'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activesupport-4.2.10/lib/active_support/dependencies.rb:240:in `load_dependency'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activesupport-4.2.10/lib/active_support/dependencies.rb:268:in `load'