Help restoring broken Gitlab instance due to missing gitlab_kas_secret file

Hello

My Omnibus Gitlab instance is currently broken following a system upgrade. Now my instance isn’t working at all (produces 500s) and whenever I attempt to run gitlab-backup create or upgrade the gitlab-ee package I get a permission denied error on file /opt/gitlab/embedded/service/gitlab-rails/.gitlab_kas_secret. This file does not exist at this location and I suspect I may need to re-create it somehow.

Has anyone seen this error? Does anyone know how I can go about fixing this?

Here’s the output generated by the package upgrade command:

$ sudo apt install --reinstall gitlab-ee
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following packages will be upgraded:
  gitlab-ee
1 to upgrade, 0 to newly install, 0 to remove and 0 not to upgrade.
3 not fully installed or removed.
Need to get 0 B/1068 MB of archives.
After this operation, 1177 kB disk space will be freed.
(Reading database ... 292371 files and directories currently installed.)
Preparing to unpack .../gitlab-ee_14.7.0-ee.0_amd64.deb ...
gitlab preinstall: Checking for unmigrated data on legacy storage
gitlab preinstall: 
gitlab preinstall: Upgrade failed. Could not check for unmigrated data on legacy storage.
gitlab preinstall: 
gitlab preinstall: rake aborted!
Errno::EACCES: Permission denied @ rb_sysopen - /opt/gitlab/embedded/service/gitlab-rails/.gitlab_kas_secret
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/jwt_authenticatable.rb:35:in `initialize'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/jwt_authenticatable.rb:35:in `open'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/jwt_authenticatable.rb:35:in `write_secret'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/kas.rb:26:in `ensure_secret!'
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/gitlab_kas_secret.rb:3:in `<top (required)>'
/opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:7:in `<top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => gitlab:storage:legacy_attachments => environment
(See full trace by running task with --trace)
gitlab preinstall: 
gitlab preinstall: If you want to skip this check, run the following command and try again:
gitlab preinstall: 
gitlab preinstall:  sudo touch /etc/gitlab/skip-unmigrated-data-check
gitlab preinstall: 
dpkg: error processing archive /var/cache/apt/archives/gitlab-ee_14.7.0-ee.0_amd64.deb (--unpack):
 new gitlab-ee package pre-installation script subprocess returned error exit status 1
Errors were encountered while processing:
 /var/cache/apt/archives/gitlab-ee_14.7.0-ee.0_amd64.deb

Oh my gosh. Something really did go awfully wrong in the last package upgrade because something somehow nuked file /opt/gitlab/embedded/service/gitlab-rails/.gitlab_kas_secret.

To resolve this all I had to do was:

/opt/gitlab/embedded/service/gitlab-rails # ln -s /var/opt/gitlab/gitlab-rails/etc/gitlab_kas_secret .gitlab_kas_secret

Now back in working state!

2 Likes

Yay :tada: thank you for sharing the resolution and glad it’s working again!