Troubleshooting external GCP secrets in Gitlab CI

GitLab CI/CD
, ,
1

See the pipeline status and CI yaml file here: Donet / Donet · GitLab

Problem to solve

Describe your question in as much detail as possible:

I am trying to troubleshoot an issue that I have with a specific job, sync-mirror. It clones the repo with --mirror and pushes to github with --mirror, but before it can it gets an SSH private key secret from GCP secrets manager. I am using an ultimate gitlab trial to use this feature.

It fetches the secret OK, but trying to use it as an SSH private key to push leads to the following error:

Error loading key "/root/.ssh/privatekey": error in libcrypto

I assumed maybe the secret value in GCP (that has newlines) was stripped into a single line, leading to an invalid private key format for OpenSSH, so I instead now store only the private key in one line in the GCP secret, and add any other header/footer of the format on the go. I verify that this can reproduce a valid private ssh key. But it still fails with the same issue. See my YAML file for more info on the exact steps taken.

I have tried to verify if the content of the secret temp file is the same as what I expect (from viewing the secret’s content in GCP secrets manager) by using sha256sum to check the hash sum without revealing the secret in the job log.

The hash sum check fails, so it looks like the secret Gitlab pulled from GCP is not the same as the exact value in plain text as I expect. How do I troubleshoot this? Is there some format that Gitlab uses to store external secrets from GCP into temp files?

Thank you for your time.

Versions

Please select whether options apply, and add the version information.

  • Self-managed
  • GitLab.com SaaS
  • Self-hosted Runners

Versions

2

I resolved the issue.

So originally, I stored an SSH private key in a GCP secret like so:

----- begin openssh private key -----
key
----- end openssh private key -----

and stored it as a file type CI variable.

Now I just store the key part of the private key file, and use it as a regular CI variable, and inject the header/footer of the key into a new file on the runner.

So in a nutshell, I am avoiding using a file type CI variable as I cannot figure out in what format it is storing my secret (plain text, if it added a newline, etc). I cannot cat a dummy test secret either, as Gitlab has a safety that masks it from the CI runner log.