SAST and allow_failure: can't get bandit to fail

I’m trying to run Gitlab’s SAST on a python project and not let the job pass if there are any findings.

All works fine, except that I can’t get allow_failure: true to be respected. I’ve tried adding allow_failure to sast, .sast-analyzer, and bandit-sast, individually and all at once, and still it passes with findings.

Am I missing anything?

Here’s how I set it up in .gtlab-ci.yml:

include:
  - template: Security/SAST.gitlab-ci.yml

sast:
  allow_failure: false
  artifacts:
    paths:
      - gl-sast-report.json

.sast-analyzer:
  allow_failure: false

bandit-sast:
  allow_failure: false

Hi @joao.pimentel1 , welcome to the GitLab Community forum! :tada:

As of GitLab 12.8 and up, All jobs that use rules: default to allow_failure: false if you do not define allow_failure:, so I’m not sure you need the allow-failure block at all.

https://docs.gitlab.com/ee/ci/yaml/#rulesallow_failure

In any case, allow_failure: false should be included under a rules heading for it to take effect.

You can safely ignore the sast and .sast-analyzer jobs, you should only need to set this for bandit-sast.

Can you try it again with this?

  - template: Security/SAST.gitlab-ci.yml

bandit-sast:
  rules:
    allow_failure: false
  artifacts:
    paths:
      - gl-sast-report.json

Hi!

I’ve tried that, but apparently that doesn’t pass on CI Lint.

I’ve tried

bandit-sast:
  rules:
    - when: always
      allow_failure: false

but still the job is passing with findings. Any idea? Thanks once again!

Hi @gitlab-greg! I still can’t make it fail with bandit coming up with findings.

Is there any chance that even if bandit returns non zero, the analyzer runner never fails?

The pass/fail for bandit indicates whether the CI job was successful, it doesn’t indicate whether or not vulnerabilities were found and reported.

The bandit-sast must be successful (Passed) for findings/results (gl-sast-report.json) to be displayed.

If the bandit-job exits with a non-zero exit code, this would indicate the job failed to produce and upload the results as gl-sast-report.json.

Can you try with:

include:
  - template: Security/SAST.gitlab-ci.yml

variables:
  SECURE_LOG_LEVEL: "debug"

bandit-sast:
  artifacts:
    paths:
      -  gl-sast-report.json

If that doesn’t work and the logs don’t indicate the job is not uploading gl-sast-report.json, I suggest testing it on a different python codebase to see if there’s something project-specific preventing this from working as expected.