Test coverage report in MR merged from different jobs

GitLab CI/CD
1

Hello! In the doc https://docs.gitlab.com/ee/user/project/merge_requests/test_coverage_visualization.html it’s said

You can specify one or more coverage reports to collect, including wildcard paths. GitLab then takes the coverage information in all the files and combines it together.

My pipeline consist of the following stages:

  • build
  • unit tests
  • integration tests

On the 'unit tests’ stage I have a job to expose a code coverage report (R1) and on the ‘integration tests’ stage I also have a job with a report (R2).

And it seems in an MR the only R1 is shown. When I expose R1+R2 from the same job on the ‘integration tests’ stage I see a merged result in an MR. But when reports come from different stages I see only the 1st one (R1).
Is that expected?.. Any way to have a merged result from different stages?

The thing is that ‘integration tests’ stage runs not always in my pipeline, so I can’t expose both the reports from the later stage.

2

Please show your .gitlab-ci.yml (at least the parts dealing with the coverage reports), it’s much easier to discuss actual code than it is to deal in abstracts.

Things I’d check:

  1. When both reports run, do they have the same structure? That is, do the paths all match up exactly so that the reports can be combined?
  2. When both reports run, do their outputs have different filenames, or is it possible one report is overwriting the other? Since the reports are going to be combined, I don’t think it’s safe to assume they’re kept separate by the analysis tool.

Despite the text you quoted, all of the examples on the page you referenced have just a single stage exposing the coverage report, with reports from previous stages being combined within the CI job(s) themselves. So it’s possible the “combine reports from multiple stages” functionality is either broken or was never implemented.

You could probably work around it yourself, though, by adding a final R3 stage that takes the report from R1, adds in the report from R2 iff it was run, and then exports the results as the final artifacts:reports:cobertura output. The Maven example looks like a pretty good template for that.

3

There are several open or incomplete issues/merges in the GitLab tracker that make me think that text about combining reports from multiple stages may be a bit optimistic. Combining them yourself and exporting only a single artifacts:reports:cobertura may be the best solution.

Make sure that the report-combining stage depends: on the prior ones, so that it’ll wait for them to finish before assembling the final report. (You’ll have to look up how to do that correctly when some prior stages may be optional, I’m not sure offhand.)

4

@FeRDNYC thank you for the reply!
here is my gitlab-ci simplified:

...

stages:
  - initialize
  - run_tests
  - integration_tests

Build:
  stage: initialize
  ...

Run Local Tests:
  stage: run_tests
  script:
    - ./gradlew runLocalTests --stacktrace
    - ./gradlew generateTestCoverageReport --stacktrace
  artifacts:
    paths:
      - ${CI_PROJECT_DIR}/build/reports/jacoco
      - ${CI_PROJECT_DIR}/*/*/reports
    when: always
    expire_in: 1d
  needs: ["Build"]

Code Coverage Visualization for Local Tests:
  stage: run_tests
  image: registry.gitlab.com/haynes/jacoco2cobertura:1.0.7
  script:
    - jacoco_paths=$(find ${CI_PROJECT_DIR} -path "**/src/main/java" -type d | sed 's/$/\//')
    - python /opt/cover2cover.py build/reports/jacoco/testCoverageReport.xml $jacoco_paths > build/reports/jacoco/localTestCobertura.xml
  needs: ["Run Local Tests"]
  dependencies:
    - Run Local Tests
  artifacts:
    paths: [build/reports/jacoco/localTestCobertura.xml]
    reports:
      cobertura: build/reports/jacoco/localTestCobertura.xml
  allow_failure: true

Run Integration tests:
  stage: integration_tests
  script:
    - ./gradlew runIntTests --stacktrace
    - ./gradlew generateIntCoverageReport --stacktrace
  artifacts:
    paths:
      - ${CI_PROJECT_DIR}/build/reports/jacoco
      - ${CI_PROJECT_DIR}/*/*/reports
    when: manual
    expire_in: 12h

Code Coverage Visualization for Int Tests:
  stage: integration_tests
  image: registry.gitlab.com/haynes/jacoco2cobertura:1.0.7
  script:
    - jacoco_paths=$(find ${CI_PROJECT_DIR} -path "**/src/main/java" -type d | sed 's/$/\//')
    - python /opt/cover2cover.py build/reports/jacoco/intCoverageReport.xml $jacoco_paths > build/reports/jacoco/intTestCobertura.xml
  needs: ["Run Integration tests"]
  dependencies:
    - Run Integration tests
  artifacts:
    paths: [build/reports/jacoco/intTestCobertura.xml]
    reports:
      cobertura:
        - build/reports/jacoco/intTestCobertura.xml
  allow_failure: true
...

Filenames are different, both the reports have the same structure. If I remove R1 stage, the R2 is displayed correctly. But once R1 is in place - R2 is not taken into the account.

Unfortunately, R3 workaround doesn’t look like a solution in my case. ‘Integration tests’ job is manual. And I would like to see a coverage report in an MR for either R1 only (in case R2 is not run at all) or for both the R1+R2 (in case int tests were run). So the R3, in that case, would depend on R1 & R2, and would never be run if R2 is also not run.

Anyway, let me check the reports structure again and issues/merges you’ve mentioned. Will update the thread in case I find anything new. Thanks

5

Yeah, I read over the code for processing coverage reports (it’s in ci/parsers/coverage/cobertura.rb), and I no longer think that would matter. It seems to reference the files only as abstract blobs, I don’t even think they have filenames anymore by the time they’ve gotten to the parser.

It still isn’t clear to me where that parser would be run on more than one file, though. Admittedly, it’s not clear to me that it’s definitely not, either. But the code there certainly appears to only process a single report input. If it’s meant to run multiple times and be aggregated, that code is clearly elsewhere.

6

I think that should be workable, personally. Here’s what you could probably do.

Since integration_tests are manual, the report assembly clearly can’t be part of that stage exclusively; you want it to run even when they’re not. So you’ll need another stage for the report processing. The examples call it visualization, so why not? (They also mention that’s a non-default stage that has to be defined, though, so keep that in mind.)

There’s an important note in the documentation on dependencies that comes into play here:

When you use dependencies , the status of the previous job is not considered. If a job fails or it’s a manual job that isn’t triggered, no error occurs.

There’s something similar for needs, using “optional”.

So, what you want is a new stage, that has dependencies on both stages that generate coverage input. And then you want to combine the reports in that final stage.

stages:
  - build
  - run_tests
  - integration_tests
  - visualize

Run Local Tests:
  stage: run_tests
  script:
    - ./gradlew runLocalTests --stacktrace
    - ./gradlew generateTestCoverageReport --stacktrace
  artifacts:
    paths:
      - ${CI_PROJECT_DIR}/build/reports/jacoco
      - ${CI_PROJECT_DIR}/*/*/reports
    when: always
    expire_in: 1d
  needs: ["Build"]

Run Integration tests:
  stage: integration_tests
  script:
    - ./gradlew runIntTests --stacktrace
    - ./gradlew generateIntCoverageReport --stacktrace
  artifacts:
    paths:
      - ${CI_PROJECT_DIR}/build/reports/jacoco
      - ${CI_PROJECT_DIR}/*/*/reports
    when: manual
    expire_in: 12h

Code Coverage Visualization:
  stage: visualization
  image: registry.gitlab.com/haynes/jacoco2cobertura:1.0.7
  script:
    - jacoco_paths=$(find ${CI_PROJECT_DIR} -path "**/src/main/java" -type d | sed 's/$/\//')
    - python /opt/cover2cover.py build/reports/jacoco/testCoverageReport.xml \
      $jacoco_paths > build/reports/jacoco/testCobertura.xml
    - if [ -f "build/reports/jacoco/intCoverageReport.xml" ]; then \
        python /opt/cover2cover.py build/reports/jacoco/intCoverageReport.xml \
        $jacoco_paths > build/reports/jacoco/intCobertura.xml; \
     fi
   - cat build/reports/jacoco/*Cobertura.xml > build/reports/jacoco/cobertura.xml
  dependencies:
    - ["Run Local Tests"]
    - ["Run Integration tests"]
  needs:
    - job: ["Run Local Tests"]
    - job: ["Run Integration Tests"]
      optional: true
  artifacts:
    reports:
      cobertura:
        - build/reports/jacoco/cobertura.xml
  allow_failure: true
...

Though you might be able to just skip the needs, with separate stages. The docs make it sound like they’ll be ordered automatically unless overridden with needs. Not 100% sure about that, though.

And if just brute-force cat-ing multiple XML files together doesn’t produce a usable file, I’m sure there’s a report-combining script out there somewhere.