Hi folks,
I am trying to get some test coverage information in my project (unfortunately on a private company repo I can not share here) and I am getting a bit confused about what does what.
I have Cobertura Reports which I marked as reports: cobertura artifacts in the CI script, but at the same time there is the
Test coverage parsing Option under Settings which seems to be a completely different thing (what exactly does this expect as input? the documentation looks where scarce to me). (How) are they related?
@OIiM - Thanks for the post!
The Cobertura report is used for the Test Coverage Visualization feature but is not currently used for the test coverage parsing to display coverage in the Merge Request widget or on a badge. That can be setup in the Settings → CI/CD or in your .gitlab-ci.yml file with the coverage keyword.
There is an open epic for future functionality using the coverage report that you can review/contribute to with your use case or to add a 
I hope this helps!
-James H, GitLab Product Manager, Verify:Testing
So what exactly should this regex match? A percentage value in the jobs text output?
@OIiM - There are several examples in the documentation depending on the language you are using to parse lines of code covered from the job logs. As long as the value is present there and the regex matches you will start to see coverage data in the jobs.
-James H, GitLab Product Manager, Verify:Testing
Yeah I have seen those examples but never any indication on what their actual goal is.
So the job-log is parsed and the regex needs to match a percentage number (with or without the percentage sign?).
Which for me means I can put the percentage into the log by parsing the cobertura summary entry and writing it to sdtout with some markers to make it unique for the regex (since as far as I am aware my current toolchain does not write any coverage info to stdout by itself).
Inclusion of the percentage sign depends on what you are parsing. In this example project I use the regex from the JaCoCo example and you can see the value parsed in the job and in the Merge Request.
I’m interested in what your use case may be beyond surfacing the data on these pages.
-James H, GitLab Product Manager, Verify:Testing
My usecase IS surfacing the data, but the way it is collected was unfortunately not very easy to understand for me since you keep finding explanations about the “xml parse” and the “log parse” way mixed when trying to find infos.
Once I understood WHAT gitlab wanted to see, matching the (line) coverage summary from a cobertura xml was easy:
/coverage line-rate="(\d*\.\d*)"/ Unfortunately the server side always interprets the stuff as percentage points even though there is no percentage sign → I had to add a small script which basically uses that match, converts to double, multiplies by 100 and only then writes the value to the log.
For powershell in case someone is interested:
$filename = $args[0]
$myline = Select-String -Path $filename -pattern 'coverage line-rate="(\d*\.\d*)"'
$mycover = Select-String -InputObject $myline.Matches[0].Value -pattern '\d+(\.\d+)?'
$coverDouble = [double]$mycover.Matches[0].Value
$coverDouble = [string](100*$coverDouble)
Write-Host "Cobertura Line Coverage is" $coverDouble"%!"Afterwards the following can finally collect the coverage from the log
coverage: /Cobertura Line Coverage is (\d*\.\d*)\%!/
@OIiM wow that’s a lot of hoops to jump through! I hope those issues I linked to help eliminate steps like this for you in the future.
-James H, GitLab Product Manager, Verify:Testing
@jheimbuck_gl Yeah, this ticket is basically what I was missing:
To be honest I could live with the regex as long as the percentage/ratio of 1 difference would be handled correctly. Unfortunately I fear that adding a check for the percentage sign could throw a lot of old pipelines off.
@OIiM great to hear that should help. I’m glad you found a solution so that cobertura report can be used for both the coverage percentage and the test coverage visualization feature.
-James H, GitLab Product Manager, Verify:Testing
Thanks for mentioning this, b/c i thought i always had to nag our org’s gitlab admin to enter the regex for me. Can you clarify which one takes precendence (gitlab settings vs. gitlab-ci.yml)?
It would be great if we could improve the gitlab documentation on these topics by adding links from and to the different pages dealing with the topic (i had to discover them individually) – while making clearer that these deal with two distinct but closely related features (one is line-by-line coverage display, while the other is the total coverage display in badges, graphs, etc.).
Your post is the first i saw that mentions all three doc pages in one sentence, that was helpful. After finally managing to set up the total coverage display, i created this issue to improve the doc to make the user journey more straightforward.
However i still haven’t managed to enable the line-by-line coverage display, but will be starting to experiment with correcting the sources / classpath bit in the cobertura xml. The hint is already there in the documentation, however i assume the examples are not straightforward and may be improved. Will report back here if i succeed.
@jheimbuck_gl i was on the verge of giving up, but i finally got line-by-line coverage display working – thanks to the comment by @rlaw_doctible below this other thread’s comment: Enabling cobertura - #15 by rodneyrehm
I had to add this script to then end our visualization step:
# replace <source> tag content with ./ (otherwise gitlab cannot map the paths to src)
- sed -i "s=<source>.*src/main/java</source>=<source>./</source>=g" target/site/cobertura.xml
This corrects the <source>-tag content in the cobertura.xml to read:
<source>./</source>
instead of
<source>src/main/java</source>
…which makes sense: if that source path is applied to the filename-path, we end up with paths starting with src/main/java/src/main/java/{code/under/test}.java and therefore gitlab cannot correctly map them.
My suggestion would be either to add the additional sed-fu to the example in the documentation, or @haynes could modify source2filename.py in their repository (which is a must-have requirement anyway in order to achieve the line-by-line coverage display) to fix the <source>-tag content as well.
Hello 
Thanks for the mention.
I think this is mostly a documentation issue. Gitlab introduced “automatic class path correction” which essentially does what the source2filename.py does.
The <source> tags are set as parameters in the cover2cover.py script. So you could simply change the script of the job to:
- 'python /opt/cover2cover.py target/site/jacoco/jacoco.xml ./src/main/java > target/site/coverage.xml'
regarding displaying the coverage in % I use the following lines in jobs that generate jacoco reports:
- 'cd target/site/jacoco/'
- awk -F"," '{ instructions += $4 + $5; covered += $5 } END { print covered, "/", instructions, " instructions covered"; print 100*covered/instructions, "% covered" }' jacoco.csv
and the following regex in the CI/CD Settings:
\d+.\d+ % covered
Wish somebody had told me that a bit earlier 
. That should go into the documentation so people don’t pull out their hair while trying to get this to work and instead get a success story straightaway. I made an issue to add it to the docs in that regards: Improve incomplete / wrong information for gathering of JaCoCo test coverage percentage (#331572) · Issues · GitLab.org / GitLab · GitLab
So you could simply change the script of the job to:
- 'python /opt/cover2cover.py target/site/jacoco/jacoco.xml ./src/main/java > target/site/coverage.xml'
I just tried that and it won’t work (neither with nor without using the source2filepath.py-script afterwards. The only way i can get line-by-line coverage display to work so far is when the <source>-tag content reads exactly ./ and the filenames are fully qualified starting with src/main/java.
Got it 
use the following line:
- 'python /opt/cover2cover.py target/site/jacoco/jacoco.xml $CI_PROJECT_DIR/src/main/java/ > target/site/coverage.xml'
The important part is adding $CI_PROJECT_DIR/ in front of src/main/java .
This is because gitlab expects source tags like this:
And the
sourcesfrom Cobertura XML with paths in the format of<CI_BUILDS_DIR>/<PROJECT_FULL_PATH>/...:
See Draft: fix: wip (!69) · Merge requests · Hannes Rosenögger / libsass-maven-plugin · GitLab for a working example.
I’ll update the documentation accordingly.
I’d like to add that the important part may also be the trailing slash at the end of the sources-path – which may have been the culprit all along.
Using that single line @haynes posted above erases the need to execute the sources2filename.py-script altogether, as the gitlab cobertura parser seems to prefix all filenames with the sources content – but a bit too literally, which i assume is the reason why it never worked w/o a trailing slash in there.
While @haynes will update the documentation for their project, I’ve opened up a pull request to fix the documentation accordingly: Fix maven & gradle examples in the docs, enabling line-by-line display of test coverage in merge requests out-of-the-box (for java & kotlin) (!62416) · Merge requests · GitLab.org / GitLab · GitLab
Very happy day, getting this solved by even less lines of CI-config code than before