210:
211: class SecretsHelper
212:
213: def self.read_gitlab_secrets
214: existing_secrets ||= Hash.new
215:
216: if File.exists?("/etc/gitlab/gitlab-secrets.json")
217>> existing_secrets = Chef::JSONCompat.from_json(File.read("/etc/gitlab/gitlab-secrets.json"))
218: end
219:
220: existing_secrets.each do |k, v|
221: if Gitlab[k]
222: v.each do |pk, p|
223: # Note: Specifiying a secret in gitlab.rb will take precendence over “gitlab-secrets.json”
224: Gitlab[k][pk] ||= p
225: end
226: else
/etc/gitlab/gitlab-secrets.json contains a lot of base64-encoded (or some other encoding, it doesn’t really matter here) stuff, but apart from that, it’s a plain text file that is readable in any program you would normally use to read text files.
GitLab does a modify of the gitlab-secrets.json file when a new secret is introduced, or if gitlab.rb specifies a direct value for any existing secret - it does so by just adding the new entry to the existing JSON content, by parsing it, and then re-writing it through JSON aware reader/writers.
You can check if your gitlab.rb was modified to add secret config values recently, and if the value in it carries any invalid characters.
However, if the entire file appears unreadable, there’s a chance some other process may have placed a file on top. What does running the file type detection tell when you run: file /etc/gitlab/gitlab-secrets.json? On the off chance that its a compressed file, the contents can be recovered by decompressing, etc.
Harsh - thanks very much for replying. Thanks for the life line.
Is the only way the gitlab-secrets.json file gets modified is if the gitlab.rb specifies a direct value for any existing secret ?
This issue we’re having is we need to upgrade our Omnibus v8.15.5, since it’s really out of date and has vulnerabilities we need to mitigate. We’d like to upgrade to v13.x.
Coincidentally, the day before we attempted to upgrade, the VM ran out of disk space and the database was having I/O errors trying to write, which may have corrupted the gitlab-secrets.json file - I’m not sure. But the entire gitlab-secrets file is not readable at this point.
After extending the filesystem to make more disk space, we ran into the error above - Chef Exceptions JSON Parse Error and in the block above where it says "Relevant File Content " it looks like it mentions the gitlab-secrets.json file. I’m guessing that the GitLab upgrade process is trying to read the gitalb-secrets file but can’t read it or parse it since if it should be human readable and it isn’t then it spits out the error above ?
If the gitlab-secrets file is definitely corrupt - which appears to be the case, the it seems like we cannot upgrade ? I don’t know for sure.
We do a have snapshot of the VM but it is about 2 years old. Obviously one of the things I’m thinking about doing reverting back to the 2 year old snapshot to get the gitlab-secrets file and then revert back to the current snapshot so we can proceed on with the upgrade as intended. But that would mean we presume that the old gitlab-secrets file of 2 years ago would be valid for our present GitLab instance.
@cdm are you running this on Linux or even UNIX?
Maybe there are some simple things to check. I am relatively novice at GitLab server administration, I have only built two; however, I have 25 years of Linux/UNIX experience.
Anyway, In the original post I see you have references to /etc/gitlab/gitlab-secrets.json .
What do you get when you execute: ls -lZ /etc/gitlab/gitlab-secrets.json ?
Can you escalate privilege to root and vi/vim the file:
sudo vi /etc/gitlab/gitlab-secrets.json
If you can, at the vi/vim (ed/ex) ‘colon prompt’ enable:
:set list
Do you see any characters that seem out of place to you? Perhaps ^M characters on the end of every (rarely a single) line?
So, consider/check
the OWNER:GROUP ownership of the file using, ls -l /etc/gitlab/gitlab-secrets.json
the SELINUX context (assuming you are running a Linux variant) with the ls -lZ /etc/gitlab/gitlab-secrets.json
the readability of all lines without “hidden” characters found by opening the file using vi/vim
I have had SSH with GitLab perform poorly due to SELINUX contexts being a problem. No guarantee that is the case this time though.
@warronf so I can vim the file but it’s just one long line (several lines on the screen) with “no end of line” character of nothing but ^@ (carrot “^” and at “@” symbols). That’s it. But thanks for responding.
Something tells me the file was copied off of a Linux/Unix system over to a Windows system, desktop or server - it does not matter, and then opened with Wordpad or Notepad and corrupted at that time. This sort of transaction and corruption is (eh) kind of common.
If you have the command dos2unix on your machine, run it against a COPY of that file.Make a copy of the original in /tmp, then run dos2unix /tmp/gitlab-secrets.json.
Use vim on the newly adjusted copy of the file - /tmp/gitlab-secrets.json and see if those “^@” strings disappear.
