Can't Recover from loss of gitlab-secrets.rb



At the beginning of December, we upgraded our GitLab system. We were upgrading from CentOS 7.5 to CentOS 7.6. As part of this upgrade, we also upgraded from GitLab 10.8.7 to 11.6.5. These migrations were effected by deploying a new instance from a new AMI, installing 10.8.7 on the new instance, restoring from the prior instance’s backups, then upgrading to 11.6.5.

It seemed like everything was fine post-upgrade. Over the nearly two months since the upgrade, no issues were reported. However, today, we received word that some users’ CI-automations were failing and that they were getting 500 errors when attempting to view their repositories’ integrations pages.

All Google searches lead to the realization “forgot to copy the gitlab-secrets.json to the the new installation”. Found a related Issue in the GitLab-CE issue tracker as well as the blurb in the documentation When the secrets file is lost.

I followed the steps. However, when I did the SELECT * FROM public."ci_variables"; step, I got zero items listed. And, when I did the DELETE FROM ci_variables; step, I got zero changes. This left me not terribly confident that the fix worked …and when I ran the reconfiguration and restart, the service returned to its prior state. So, I’m at a bit of a loss of how to resolve my current situation.