How to fail job and pipeline if there are failed tests in JUnit report?

How to Use GitLab
,
1

Hi.
My project (mobile application) has pipeline with stage for running UI tests (Appium) based on TestNG and maven.
I set up displaying JUnit reports in pipeline results as artifacts
artifacts:reports:junit: - surefire-reports/TEST-*.xml
But even if there are failed tests in report, job is still marked as succeeded, and pipeline is green.
So is it possible to fail pipeline if report contains failed tests?

2

@mikamika - Thanks for posting!

By default if you have a failing test the job and pipeline should fail. You can see an example in a small project here.

There is a way to override this behavior by using rules:allow_failure in the job. Documentation for that is here.

Good luck!

-James - Product Manager GitLab

1 Like
3

Hi @jheimbuck_gl Thanks for swift response. But it does not work.

There is job for running maven tests (mvn test). It works inside docker container.

One test is failed by assertion. In maven log inside job log I see
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0
[INFO] BUILD FAILURE

Then, there is artefacts uploading:
Uploading artifacts to coordinator… ok

But at the end of job log is:
Job succeeded

So, job is marked as succeeded, and all pipeline is succeeded too.

May there is additional option somewhere in Gitlab that I forgot? The only thing I set up is .gitlab-ci.yml file with UI test job stage (in addition for stages for building app etc.)

4

GitLab CI doesn’t look at the JUnit reports to determine if the job failed. It looks at the status code returned from the commands in the job. If you’re running your tests from something that swallows up failure status codes, perhaps a wrapper shell script, then the CI will not know that they failed, regardless of what the report says.

1 Like
5

@xenomachina - you’re totally correct on that, it’s based on job exit codes not test results.

@mikamika - in my example i’m running the tests as part of the build job and when the tests fail the job fails. If the build job was passing but tests failing I would likely setup another step to parse the resulting test results and if a failure is found exit with a failure. I did something similar when code coverage decreases with an MR in this project. That’s not an ideal solution by any means but hopefully helps keep you moving.

-James H - GitLab Product Manager

2 Likes
6

Sorry for reviving a really old thread, but I just started using GitLab and had to spend a lot of time before realizing that GitLab doesn’t fail a pipeline with test failures. Seems like a weird choice by GitLab but it is what it is.

My only contribution right now is that I came here to say that I haven’t found any good tools for checking junit xml and emit an error code, but I did put together this little hack to make it work for my case.

lint-job:
  stage: build
  script:
    # Your build steps that create the JUnit XML
    - ./do_build.sh
    # Horrible way of parsing XML
    - test -f lint.xml && grep -L "<failure" lint.xml

Hopefully someone else will find it and not waste as much time as I did.

1 Like
7

I have used the same but not working.Is there any other solution ?

8

I am currently using this script

check.sh:

#!/bin/bash

if [ -z $1 ]; then
  echo "Usage: $0 [path/to/file.xml]"
  exit 1
fi

if ! test -f $1; then
  echo "FILE NOT FOUND: $1"
  exit 1
fi

if grep "<failure" $1; then
  # Grep found line = error
  echo "LINT ERROR in $1"
  exit 1
fi

then in .gitlab-ci.yml

  script:
    - ./check.sh test/build/test-results.xml
10

Thanks Erik for your suggestion.
This script is working fine.

11

I’ll likely end up using this, but having this feature be an option would be a big boon.

For instance, I’ve got tests the are “Inconclusive” the build settings aren’t configured for them. Unity returns a bad exit code whenever not all tests pass, so I have to resort to some janky behavior to work around this.

12

On windows you could use Powershell Select-xml query to validate xml attributes failure & errors

stages:          # List of stages for jobs, and their order of execution
  - test

unit-test-job:   # This job runs in the test stage.
  stage: test    
  rules:
    - changes:
      - Tests/TestResults.xml
      allow_failure: true
  artifacts:
    when: always
    paths:
      - Tests/TestResults.xml
    reports:
      junit: Tests/TestResults.xml
  script:
    - echo "Running unit tests... "
    - Select-Xml -Path Tests\\TestResults.xml -XPath '/testsuites' | ForEach-Object { if (($_.Node.failures -gt 0) -or ($_.Node.errors -gt 0)) {exit -1} }