Before_script commands intermittently fail

My build script succeeds about 75% of the time, but intermittently fails when seemingly nothing has changed, and selecting “retry” on the job almost always succeeds on the first attempt. You can see in the before_script it takes an envvar and echos it into a .json file which gcloud requires to authorize access.

I posted a successful output and a failed output, and I can’t figure out what might be happening. Do before_script tasks run in parallel so sometimes the echo command hasn’t run before gcloud auth has? Maybe echo is finishing slowly enough gcloud auth runs before the previous command has finished writing the env var to the .json?

Intermittent problems are always the hardest to debug, any ideas would be great!

.gitlab-ci.yml

before_script:
  - echo ${CREDS} > ${CREDS_FILE}
  - gcloud auth activate-service-account --key-file=${CREDS_FILE}
  - gcloud config set project ${GCLOUD_PROJECT_ID}

successful:

Checking cache for default...
Successfully extracted cache
$ echo ${CREDS} > ${CREDS_FILE}
$ gcloud auth activate-service-account --key-file=${CREDS_FILE}
Activated service account credentials for: [gitlab@my.iam.gserviceaccount.com]
$ gcloud config set project ${GCLOUD_PROJECT_ID}
Updated property [app/project].
$ make build

failed:

Checking cache for default...
Successfully extracted cache
$ echo ${CREDS} > ${CREDS_FILE}
$ gcloud auth activate-service-account --key-file=${CREDS_FILE}
ERROR: (gcloud.auth.activate-service-account) Could not read json file /creds.json: No JSON object could be decoded
ERROR: Job failed: exit code 1
1 Like

This exact issue is happening to me.
It doesn’t always work on retry.