Cobertura coverage not showing in diff or jobs

I am trying to get test coverage visualization for our monorepo as described here Test Coverage Visualization

We are using a self-managed gitlab runner with the docker+machine executor hosted on AWS EC2 instances. The job from the gitlab-ci.yml is below

sdk:
  stage: test
  needs: []
  artifacts:
    when: always
    reports:
      cobertura: /builds/vividcloud/autoguide/amr/sdk/coverage/cobertura-coverage.xml
  <<: *main
  <<: *tests
  <<: *rules
  <<: *tags

The artifact gets saved just fine and looks normal when I download it, but I don’t get any visualization as described in the documentation linked above. I just updated the gitlab runner to V14.0.0 thinking that might be the problem, it’s not.

I have spent nearly 2 days trying to get this simple task to work. I see in posts that I have searched that there are some feature flags to enable to get this to work, but from what I can tell they are no longer needed. I don’t have any sort of regex pattern setup as from my understanding that is only for printing the coverage info to stdout.

I’m sure I am overlooking something trivial and I really need a sanity check here as I have already spent way more time on this than I can afford.

Thanks in advance for any help!

Hi @Btripp1986 thanks for the post!

I just wanted to make sure you’re looking on the Changes Tab of the MR and the tests ran/cobertura.xml file was created as part of that MR’s latest pipeline? I should also call out there there have been fixed for automatic path correction in the 13.8 and 13.9 versions so if you are not running that on a self managed issue there could be problems with the paths.

Hopefully this helps!

-James H, GitLab Product Manager, Verify:Testing

1 Like

Thanks for the reply @jheimbuck_gl

I am in fact not using self-hosted gitlab, and I am looking at the changes tab of the MR. I double checked and the cobertura-coverage.xml is getting generated in that MR pipeline.

After looking through the link you provided on automatic path correction I think the cobertura-coverage.xml file might be the issue. Since we have a monorepo for this project the pipeline jobs are running on different directories of the repository (by that I mean the test pipeline has a job for running the tests in $PROJECT_ROOT/feature1, another job in $PROJECT_ROOT/feature2, etc. Because of this (and the fact that the test files are no in the root dir) the .xml generated shows the following

<source>/builds/mycompany/mygroup/myproject/feature1</source>
...
...
<class name="Banner.tsx" filename="src/components/Banner/Banner.tsx" line-rate="0.23079999999999998" branch-rate="0">

From my understanding of the documentation you linked, I think gitlab is expecting

<source>/builds/mycompany/mygroup/myproject</source>
...
...
<class name="Banner.tsx" filename="feature1/src/components/Banner/Banner.tsx" line-rate="0.23079999999999998" branch-rate="0">

I am going to attempt modifying these values in the job script to match the latter output. Please advise if you can think of anything else. Thank you for your help!

1 Like

@Btripp1986 I think you’re on the right track with that approach.

-James H, GitLab Product Manager, Verify:Testing

1 Like

@jheimbuck_gl I have added an after script containing the following

- sed -i 's;filename=";filename="feature1/;g'
- sed -i 's;myproject/feature1;myproject;g'

I have downloaded the report artifact and it looks right now

<source>/builds/mycompany/mygroup/myrepo</source>
...
...
<class name="ThemeProvider.tsx" filename="sdk/src/contexts/ThemeProvider.tsx" line-rate="1" branch-rate="1">

I am still not seeing code overage in the MR diff view, I am also still not seeing a code coverage % in the CI => Jobs page either.

If you have any ideas or a course of action you would recommend please let me know as I am running out of ideas. I was considering opening an issue, but I want to make 100% sure that this isn’t a user error before I do. Thank you for your guidance.

@Btripp1986 - Thanks for the update!

I’ll poke around a bit with the team to see what ideas they have about your latest .

For code coverage on the Jobs page you’ll need to parse the coverage which you can find information about in the documentation.

-James H, GitLab Product Manager, Verify:Testing

1 Like

Hi @Btripp1986! I noticed filename="sdk/src/contexts/ThemeProvider.tsx" from the sed modified XML. Just to confirm, is it correct that the full path (relative to the project root) of the class ThemeProvider.tsx, is sdk/src/contexts/ThemeProvider.tsx which includes sdk/?

-Erick B, Senior Backend Engineer, Verify:Testing

Thanks for looking at this.

Yes, the full path to ThemeProvider.tsx is <project-root>/sdk/src/contexts/ThemeProvider.tsx

After reading @jheimbuck_gl comment, I know I do not have the RegEx patter correct, so I assume that is the reason for coverage % not showing in the Jobs page. Would this affect the visualization in the MR diff, or is that feature unconcerned with the RegEx pattern?

No it won’t affect the MR diff visualization. They rely on 2 different artifacts.

When on the MR diff page, can you please check in your browser’s console if it made any requests to something like .../-/merge_requests/:mr_id/coverage_reports.json? And if it did, can you check if the JSON response contained any map of filenames?

Sorry for the late response @iamricecake

My team has changed its git strategy and we are now committing directly to the default branch, so coverage visualization is no longer required for us(no MRs).

I did get the coverage % to show up in the jobs page, which will be enough for our purposes.

If we change our git strategy again I will be implementing the coverage visualization, in which case I will refer back to this thread.

I thank you and James for your help. You folks are great and I preach your product to everyone I know.

1 Like

Glad @Btripp1986 & @iamricecake posted on this topic. I’m running a self-hosted 13.12.3 setup. I’ve got pipelines uploading cobertura artifacts (I can download them ok). Am also working through how to get the visualization working in the MR pages. Checking the console when loading the diffs page, I see the GET request for https://gitlab…stevens.edu/rt…/ioif/-/merge_requests/32/coverage_reports.json with response SC 204. But Chrome reports the resource size of 0 (the content-length header not shown). Suggests to me that whatever processing is returning the payload of the request has determined no content should be provided.

Any chance this is a function of the paths defined in the , <class filename=> attributes not being correct? [FYI: I now have a sed script to do as Ben indicated to ‘massage’ these paths but not used on this particular set of uploaded artifacts] - as these cobertura formated coverage xml files are created by the pytest-cov plugin during a pytest launched by tox, so the paths are tox testenv locations.

Looking at the log files generated in our installation both at upload time and this morning when trying to get the MR diffs visualization working, I don’t see any details revealing any issues with retrieving, using the coverage_reports.json.

What log file(s) might reveal issues with generating this coverage_reports.json payload?
MTIA for any feedback, insights.

some more details: the file downloaded via the ‘Download tests:cobertura artifacts’ menu choice in the pipeline’s Artifacts menu is the aggregation of the 3 different cobertura formatted coverage reports created, uploaded by the pipeline’s tests job. And in my Chrome console, there is a 2nd GET request for the coverage_reports.json doc. And this request’s response has a json object. But there are only 2 files with coverage lines, counts being returned. In the downloaded coverage XML, there are other elements in the same instance. Which have the same valid paths [1 out of 3 uploaded coverage XML files have tox testenv paths, but not this one so, as indicated by the Gitlab docs on auto path correction, the source files are ‘discoverable’ by the coverage paths. No idea yet as to why the other elements not being in the response.

The other 2 instances have no filename instances returned in the response body. Which is possibly explained by these 2 instances having incorrect tox testenv paths:
<class branch-rate="0" complexity="0" filename=".tox/py37-graphdb/lib/python3.7/site-packages/ioif_core/Exceptions.py"

response JSON: (long sequences removed)

{"files":{"app/svc/ioif/ioif_svc/my_defaults.py":{"1":1,"8":1,"9":1,...,"37":1},"app/svc/ioif/swagger_server/__main__.py":{"3":0,"4":0,"5":0,..."92":0,"93":0}}}

Hi, I am also facing the same issue where the coverage percentage shows up just fine on the jobs page, but I don’t see anything on the MR diff page

Hi everyone and hi @iamricecake :raised_hand: :slight_smile:

I’m also facing the same issue whereas I get the coverage badge with the right percentage from this report.
The part of my gitlab-ci with coverage :

    - coverage run manage.py test --settings='settings'
    - coverage report
    - coverage xml -o coverage.xml
  coverage: '/^TOTAL.*\s+(\d+.\d+%)$/'
  artifacts:
    reports:
        coverage_report:
          coverage_format: cobertura
          path: coverage.xml

The coverage report is available to download and I’m able to view the coverage percentage in each MR.

Do you have any idea please ?
Thanks a lot

Fred

Hey, @SteveHespelt, have you had any luck figuring this out? I am on the same version of self-hosted gitlab as you are, and cannot for the life of me figure out how to get code coverage working. I’ve tried everything from massaging the paths in the cobertura xml files to enabling a feature flag. Nothing seems to work.

@jheimbuck_gl
Have you got any feedback from the team ?

I have the same issue as others here in this thread. Coverage percentage is shown but the MR diff doesn’t show coverage.

Using gcovr to output cobertura formatted xml
Running gitlab 14.10.5

@soped

I don’t know of anything that would have changed with this feature but have tagged the team internally who works on this now.

-James H, GitLab Product Manager, Verify:Pipeline Execution

Hi @aronweiler, I’ve not. I resorted to downloading the htmlcov artifacts I generate via pytest --cov-report=html:{envtmpdir}{/}htmlcov at the same time I’m generating the Cobertura xml file via --cov-report=xml:test_reports/coverage.xml in my tox usage. Tox commands entry then zips up the htmlcov directory and mv the zip over to where the artifacts.paths entry for the job’s zip files. Better than nothing, my boss likes to review those occasionally.

Thanks @jheimbuck_gl

As I understand it the coverage report is processed in a background job when pipeline is done. Can I somehow see the outcome of this background job ?

@soped I don’t believe that is available in the UI today no.

-James H, GitLab Product Manager, Verify:Pipeline Execution

I am using on prem gitlab v16.04. I have set up pipeline as below

# .gitlab-ci.yml

image: node:12

# Define the stages of the pipeline
stages:
  - test

cache:
  paths:
    - node_modules/

# Define the job that will run tests
test:
  stage: test
  script:
    - npm install
    - npm run test -- --coverage --verbose --coverageReporters=cobertura
  artifacts:
    reports:
      coverage_report:
        coverage_format: cobertura
        path: coverage/cobertura-coverage.xml
  # Add coverage to code coverage statistics
  coverage: '/All files[^|]*\|[^|]*\s+([\d\.]+)/'

  # Define when this job should be executed
  rules:
    - if: '$CI_PIPELINE_SOURCE == "merge_request_event" && $CI_MERGE_REQUEST_TARGET_BRANCH_NAME == "master"'


I am trying to get test coverage visualizer in diff view of the merge request working. but i cannot see it. I can see the artifact for each job in the pipeline when run. I can see the cobertura-coverage.xml file in the artifact