When performing an upgrade and therefore running reconfigure, and on all future runs of
gitlab-ctl reconfigure the permissions of the
.gitlab-workhorse-secret are changed to
root:root, which means that gitlab-rails cannot open the file in order to read them and use them. This causes it to fail to migrate, during an upgrade, or fail to boot during normal operation. I can see from the chef recipe that it is designed to own it to root in every case, which is what I worked around in order to get us up and running.
I believe I caused the problem by specifying
gitlab_workhorse['secret_token'] = 'redacted'
in our gitlab.rb.erb, but i’m not sure how.
I’m fairly sure this is self-inflicted pain, but it’s unclear to me how.
Edit: The exception is lost in my terminal scrollback, but it was an ENOACCESS in the secret reader that has a rescue
write_secret block. Both branches of that code gave ENOACCESS, which makes sense.
Edit: An wild exception appears:
This occurred immediately after installing GitLab 11.3.1 via the gitlab-ci dpkg. Note this time, the upgrade worked. I assume because there were no migrations and no database hooks involved, therefore the app didn’t need to be loaded during the gitlab-ctl reconfigure post install dpkg hook.
==> /var/log/gitlab/unicorn/unicorn_stderr.log <== Errno::EACCES: Permission denied @ rb_sysopen - /opt/gitlab/embedded/service/gitlab-rails/.gitlab_workhorse_secret /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/workhorse.rb:180:in `initialize' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/workhorse.rb:180:in `open' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/workhorse.rb:180:in `write_secret' /opt/gitlab/embedded/service/gitlab-rails/config/initializers/gitlab_workhorse_secret.rb:4:in `rescue in <top (required)>' /opt/gitlab/embedded/service/gitlab-rails/config/initializers/gitlab_workhorse_secret.rb:1:in `<top (required)>' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activesupport-4.2.10/lib/active_support/dependencies.rb:268:in `load' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activesupport-4.2.10/lib/active_support/dependencies.rb:268:in `block in load' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activesupport-4.2.10/lib/active_support/dependencies.rb:240:in `load_dependency' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activesupport-4.2.10/lib/active_support/dependencies.rb:268:in `load'