Error reading gitlab-secrets

I moved my gitlab installation to a bigger hard drive by mounting a new drive on /var/opt/gitlab, and moving all of the files to the new drive. I made sure to get all of the hidden files. Now most things are working OK, however users cannot use their SSH keys. I tried to run gitlab-ctl reconfigure, but I get an error reading /etc/gitlab/gitlab-secrets.json. That file was not touched during the update. I suspect that the file was bad before the upgrade, but since we rarely restart the service we didn’t realize it until the upgrade forced a reboot. I do have 7 days of backups of the secrets file, but all of them have the same problem.

Since the error says that the file has an invalid character in json, is there any way to open and read it outside of gitlab to find and remove the bad character? I’ve seen similar errors and the theory has been that the file was incorrectly edited (Gitlab-secrets.json - is this file suppost to be readable in a text editor? - #6 by warronf), however that definitely did not happen here. I am the only user with server root access. What’s the best path to recovery?

Recipe Compile Error in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab-ee/recipes/default.rb


lexical error: invalid char in json text.

36: existing_secrets = {}
38>> existing_secrets = Chef::JSONCompat.from_json("/etc/gitlab/gitlab-secrets.json")) if File.exist?("/etc/gitlab/gitlab-secrets.json")
40: existing_secrets

As a follow-up, I removed the offending gitlab-secrets and ran gitlab-ctl reconfigure to confirm that is the only problem, and it ran successfully and created a new gitlab-secrets.json. Now I see that a new version of that file is completely normal looking json. The version I have in all my backups looks like it is binary or encoded somehow. Any ideas for how the file became something other than json in all my recent backups, even though gitlab has been running? It looks like I might be on the path of recovering as if I did not have a backup, is that correct?