Gitlab not passing CI/CD env vars to Auto Devops Kubernetes

Around 7pm last night our API stopped working. Research indicates it no longer has the right credentials for our database. Further investigation reveals all is fine locally, and it is only our live service that is down.

Come to find that it doesn’t appear ENV vars are being properly passed from CI/CD settings, no K8S_SECRET_VARIALBE is being used.

We also notice this:

Downloading postgresql from repo https://kubernetes-charts.storage.googleapis.com/

Despite POSTGRES_ENABLED = false being set at the account level.

This is a clear indication to me that Auto Devops builds are failing to load ENV vars properly.

Solved this… user error, sorry Gitlab. Here is the problem I ran into:

Coding along last night, rewrite some history with git commit --amend, try to push to master and get told protected branch! What?! remove that rule! … little did I realize, it is a toggle “Protected” variable, tied to several of my CI/CD ENV variables that I had set without understanding what it did. As soon as I removed the protected master branch, several of our secrets no longer passed into Kubernetes during the build. I unchecked the protected toggle and on each variable and we’re back in business.