Fail the pipeline if Secret Detection finds vulnerabilities

Much to my surprise the Secret Detection documentation does not suggest ways to have the build fail if the secret_detection job finds vulnerabilities. Such “post-processing” is particularly important for all the folks not on the Ultimate tier (the majority?) as you have no security dashboard, no security tab for the pipeline and no security widget for MRs.

Are we expected to add after_script commands (or a subsequent job) to parse the gl-secret-detection-report.json and force-fail the job by e.g. exit 42? Curious to find out how others addressed this.

<starts-hacking-pipeline-files>

Arggh…due to No possibiity to fail the job in after_script (#21008) · Issues · GitLab.org / GitLab · GitLab we can’t fail the job in after_script, sigh. This is my WIP

  after_script:
    - |
      apk add --no-cache jq
      reportFile="gl-secret-detection-report.json"
      cat "$reportFile"
      if [ "$(jq ".vulnerabilities | length" $reportFile)" -gt 0 ]; then
              echo "Vulnerabilities detected. Please analyze the report in $reportFile."
              exit 42
      fi
1 Like

By using an entirely different stage to do the validation in a proper script: section. Pass the artifact between stages.

like

  stage: validate
  variables:
    GIT_STRATEGY: none
  before_script:
    - apk add --no-cache jq
  script:
    - jq -e ".vulnerabilities | length == 0" gl-container-scanning-report.json
1 Like

Yep, it’s the obvious work-around I have now in place but it’s far from ideal IMO.

Are we expected to add after_script commands (or a subsequent job) …

To wrap this up here’s the pipeline definition we currently use to fail on detected vulnerabilities:

include:
  - template: Security/Secret-Detection.gitlab-ci.yml

  # defined here https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Jobs/Secret-Detection.gitlab-ci.yml#L21
  SECRET_DETECTION_REPORT_FILE: "gl-secret-detection-report.json"
  SECURE_LOG_LEVEL: "debug"

stages:
...
  - secret-detection
  - secret-detection-eval
...

# inherits from here https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Jobs/Secret-Detection.gitlab-ci.yml
# as imported above
secret_detection:
  stage: secret-detection
  artifacts:
    paths:
      # in the "parent" job this file is declared as a report artifact, but
      # we also need it as a regular artifact for the subsequent job
      - $SECRET_DETECTION_REPORT_FILE
    expire_in: 1 hour

evaluation:
  stage: secret-detection-eval
  variables:
    # this job only requires the $SECRET_DETECTION_REPORT_FILE
    GIT_STRATEGY: none
  cache: {}
  before_script:
    - apk add --no-cache jq
  script:
    # check if '{ "vulnerabilities": [], ..' is empty in the report file if it exists
    - |
      if [ -f "$SECRET_DETECTION_REPORT_FILE" ]; then
        if [ "$(jq ".vulnerabilities | length" $SECRET_DETECTION_REPORT_FILE)" -gt 0 ]; then
          echo "Vulnerabilities detected. Please analyze the artifact $SECRET_DETECTION_REPORT_FILE produced by the 'secret-detection' job."
          exit 80
        fi
      else
        echo "Artifact $SECRET_DETECTION_REPORT_FILE does not exist. The 'secret-detection' job likely didn't create one. Hence, no evaluation can be performed."
      fi

3 Likes

In trying this solution, I continue to get this error in my secret_detection job:

Uploading artifacts for successful job  

Uploading artifacts...
WARNING: : no matching files                      
ERROR: No files to upload                          
Uploading artifacts...
gl-secret-detection-report.json: found 1 matching files and directories `

Which causes the -f check to always fail.

Bumping this one. I tried marcelstoer’s jobs above and am also seeing the failure to upload.

Additionally, I am unclear on what image is needed for the secret-detection-eval job. I am guessing any that has apk available.

I should have waited just a bit to post as I found a fix for this. For me at least the variables here were not being inherited in the jobs:

  - template: Security/Secret-Detection.gitlab-ci.yml

  # defined here https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Jobs/Secret-Detection.gitlab-ci.yml#L21
  SECRET_DETECTION_REPORT_FILE: "gl-secret-detection-report.json"
  SECURE_LOG_LEVEL: "debug"

I verified in the ci editor/merged yaml interface that this was the case. I simply added the variable SECRET_DETECTION_REPORT_FILE: "gl-secret-detection-report.json" to both jobs and it is uploading correctly now. We are on SaaS GitLab, not self-hosted so this may be an operational difference in versions.

So, thanks to marcelstoer for the process and I hope this helps ctote1.

I’ve been able to use the solution from @marcelstoer with success, but I note that I can achieve the desired outcome (pipeline fails upon detection of secrets) within the same job.

What is the reasoning behind having the validation as a separate job? What might I be losing / problems am I creating for myself by doing it within the same job?

variables:
  SECRET_DETECTION_REPORT_FILE: "gl-secret-detection-report.json"

...

stages:
- test
- deploy

secret_detection:
  allow_failure: false
  rules:
    - if: $SECRET_DETECTION_DISABLED
      when: never
    - if: '$CI_COMMIT_BRANCH || $CI_COMMIT_TAG'
  before_script:
    - apk add --no-cache jq
  script:
    - /analyzer run
    # check if '{ "vulnerabilities": [], ..' is empty in the report file if it exists
    - |
      if [ -f "$SECRET_DETECTION_REPORT_FILE" ]; then
        if [ "$(jq ".vulnerabilities | length" $SECRET_DETECTION_REPORT_FILE)" -gt 0 ]; then
          echo "Vulnerabilities detected. Please analyze the artifact $SECRET_DETECTION_REPORT_FILE produced by the 'secret-detection' job."
          exit 80
        fi
      else
        echo "Artifact $SECRET_DETECTION_REPORT_FILE does not exist. The 'secret-detection' job likely didn't create one. Hence, no evaluation can be performed."
      fi

sast:
  stage: test

include:
- template: Security/SAST.gitlab-ci.yml
- template: Security/Secret-Detection.gitlab-ci.yml

By overwriting secret_detection#script you are effectively disconnecting yourself from the GitLab template you include. If they ever change their implementation your job might either work incorrectly or fail completely. There is no guarantee the command you need to run is always going to be /analyzer run. So, the problem with this approach is that it’s just not future-proof.

1 Like

it’s just not future-proof

Indeed, it is not. If the runner ever uses a different package manager, then the jq install step would also break.

Also wasteful to keep reinstalling jq so frequently.

And we have to redo the code snippet again for each type of GitLab scanner.

More good reasons that GitLab should fix the semantic to fail the stage in the event of a finding.