Around 7pm last night our API stopped working. Research indicates it no longer has the right credentials for our database. Further investigation reveals all is fine locally, and it is only our live service that is down.
Come to find that it doesn’t appear ENV vars are being properly passed from CI/CD settings, no K8S_SECRET_VARIALBE is being used.
We also notice this:
Downloading postgresql from repo https://kubernetes-charts.storage.googleapis.com/
POSTGRES_ENABLED = false being set at the account level.
This is a clear indication to me that Auto Devops builds are failing to load ENV vars properly.