My project (mobile application) has pipeline with stage for running UI tests (Appium) based on TestNG and maven.
I set up displaying JUnit reports in pipeline results as artifacts
artifacts:reports:junit: - surefire-reports/TEST-*.xml
But even if there are failed tests in report, job is still marked as succeeded, and pipeline is green.
So is it possible to fail pipeline if report contains failed tests?
@mikamika - Thanks for posting!
By default if you have a failing test the job and pipeline should fail. You can see an example in a small project here.
There is a way to override this behavior by using
rules:allow_failure in the job. Documentation for that is here.
-James - Product Manager GitLab
Hi @jheimbuck_gl Thanks for swift response. But it does not work.
There is job for running maven tests (mvn test). It works inside docker container.
One test is failed by assertion. In maven log inside job log I see
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0
[INFO] BUILD FAILURE
Then, there is artefacts uploading:
Uploading artifacts to coordinator… ok
But at the end of job log is:
So, job is marked as succeeded, and all pipeline is succeeded too.
May there is additional option somewhere in Gitlab that I forgot? The only thing I set up is .gitlab-ci.yml file with UI test job stage (in addition for stages for building app etc.)
GitLab CI doesn’t look at the JUnit reports to determine if the job failed. It looks at the status code returned from the commands in the job. If you’re running your tests from something that swallows up failure status codes, perhaps a wrapper shell script, then the CI will not know that they failed, regardless of what the report says.
@xenomachina - you’re totally correct on that, it’s based on job exit codes not test results.
@mikamika - in my example i’m running the tests as part of the build job and when the tests fail the job fails. If the build job was passing but tests failing I would likely setup another step to parse the resulting test results and if a failure is found exit with a failure. I did something similar when code coverage decreases with an MR in this project. That’s not an ideal solution by any means but hopefully helps keep you moving.
-James H - GitLab Product Manager
Sorry for reviving a really old thread, but I just started using GitLab and had to spend a lot of time before realizing that GitLab doesn’t fail a pipeline with test failures. Seems like a weird choice by GitLab but it is what it is.
My only contribution right now is that I came here to say that I haven’t found any good tools for checking junit xml and emit an error code, but I did put together this little hack to make it work for my case.
lint-job: stage: build script: # Your build steps that create the JUnit XML - ./do_build.sh # Horrible way of parsing XML - test -f lint.xml && grep -L "<failure" lint.xml
Hopefully someone else will find it and not waste as much time as I did.