Getting pytest tests and coverage from commits when separate MR pipeline

GitLab CI/CD
1

We had a working CI testing pipeline on gitlab.com that reported our unit test results and code coverage properly in our merge requests, setup using something like the following:

pytest:
  stage: test
  before_script:
    - pip install numpy scipy pandas xarray pytest pytest-cov netCDF4
  script:
    - pytest --cov --cov-report term --cov-report xml:coverage.xml --junitxml=report.xml --cov-config=.coveragerc
  coverage: '/(?i)total.*? (100(?:\.0+)?\%|[1-9]?\d(?:\.\d+)?\%)$/'
  artifacts:
    when: always
    paths:
      - report.xml
    reports:
      junit: report.xml
      coverage_report:
        coverage_format: cobertura
        path: coverage.xml

We recently added a more detailed data-based reporting CI job that uses quarto notebooks to run and compare different versions of our repo. Because this takes a fair bit of processing, we only want this to run on merge requests, so we set it up like the following:

sensitivity_report:
  stage: deploy
  rules:
    - if: $CI_PIPELINE_SOURCE == 'merge_request_event'    
  before_script:
    # snipped code installing quarto and associated libraries
  script:
    # snipped code using quarto to render a particular notebook
  artifacts:
    name: ${CI_COMMIT_REF_NAME}_sensitivity_report
    paths:
      - ${CI_COMMIT_REF_NAME}_sensitivity_report.*

Now, the pytest job is running happily on any commit, and the sensitity_report is only running on merge requests. Everything seems good, but the merge request Overview page is now only showing the outcome of the sensitivity report, and the unit test pass rate and coverage is not shown:

image
image1166×168 21.7 KB

For comparison, this is what we used to see on a merge request before we added the sensitivity test:

image
image1177×255 27.5 KB

Both jobs are being run though, as you can see here in the pipelines of the commit itself:

image
image1606×729 51.2 KB

It feels like because the merge request is splitting the jobs into two seperate pipelines - one for the commit and one for the MR - it is only looking in the MR pipelines for testing results, and because the test is in the commit it isn’t showing them here. This isn’t great, because we want it to be clear in our MR when the testing is passing or not, and that our code coverage is adequate.

I would appreciate any ideas on how I can get our merge-request overview page to collect these pytest results and show them like it did before we added the sensitivity report job.

2

I ended up adding another pytest job that explicity runs in the MR pipeline, and this seems to have worked:

# run pytest on every commit
pytest_commit:
  stage: test
  before_script:
    - pip install numpy scipy pandas xarray pytest pytest-cov netCDF4
  script:
    - pytest --cov --cov-report term --cov-report xml:coverage.xml --junitxml=report.xml --cov-config=.coveragerc
  coverage: '/(?i)total.*? (100(?:\.0+)?\%|[1-9]?\d(?:\.\d+)?\%)$/'
  artifacts:
    when: always
    paths:
      - report.xml
    reports:
      junit: report.xml
      coverage_report:
        coverage_format: cobertura
        path: coverage.xml

# explicity run pytest for merge request commits (again), to ensure that they 
# are available in the merge request pipeline
pytest_merge_request:
  extends: pytest_commit
  rules:
    - if: $CI_PIPELINE_SOURCE == 'merge_request_event'

It does now run pytest twice for every commit in a merge request, so I’d still welcome feedback on ways to do this better.

3

Hey David,

The image you’ve shown above from pipelines, are two different pipelines - AFAIK your jobs should be part of the same pipeline to show up on MRs (normally either branch pipeline or MR pipeline).

I’d suggest enabling MR pipeline as stated in the docs to switch between branch pipeline and MR pipeline.

So, in addition to that workflow statement from the docs (that defines how your whole pipeline runs), you can add to your pytest job to run on branches AND merge requests, e.g. with only:

pytest:
  stage: test
  # here comes the rest of it
  only:
    - branches
    - merge_requests

and for the sensitivity:

sensitivity_report:
  stage: deploy
  # blah blah
  only:
    - merge_requests

This will make your pytest run on every push, and sensitivity report additionally on MR - and you should be able to see all your reports in MR.

Note: I am assuming here you don’t need pipelines on tags, or triggered from any other source - if you do, you will have to add them to the workflows section too (and to the only/rules in the jobs as you wish), otherwise they won’t be able to run. At least from my experience :smiley:

Hope this helps or at least gives you further ideas for your implementation :slight_smile:

4

Thanks @paula.kokic, I added this to the top of my file:

# setup workflow rules so that branch pipelines run on every commit, merge
# request pipelines run on every merge request, but that branch pipelines
# don't run when there is an open merge request
workflow:
  rules:
    - if: $CI_PIPELINE_SOURCE == 'merge_request_event'
    - if: $CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS
      when: never
    - if: $CI_COMMIT_BRANCH

Went back to the single pytest job and it all works well now.

It looks like only clauses are deprecated now, but that aspect was already covered adequately by the rules I already had on the jobs. Now it looks like pytest will run as a branch pipeline on every branch commit, except those in a merge request, and the sensitvity report and pytest will run in the same pipeline for merge request.

Thanks for your help and the pointers to the solution!

1 Like