CI using only:changes is running for incorrect jobs after recent gitlab.com upgrades

I have a mono repo that I’m trying to use Gitlab with, and I’m really struggling to understand the gitlab logic as to when a job will run as the changes:only element doesn’t seem to work reliably.

In my root folder I have the following file that refers to three sub-projects I want to build:

stages:
  - install
  - test
  - build
  - deploy

include:
  - api/asset/.gitlab-ci.yml
  - mobile/.gitlab-ci.yml
  - webman/.gitlab-ci.yml

In the webman folder I have a .gitlab-ci.yml file that has the following job:

web_test:
  stage: test
  image: node:12.16-alpine
  only:
    changes:
      - webman/**
      - webman/**/**
      - webman/.gitlab-ci.yml
  before_script:
    - cd webman
    - pwd
  script:
    - pwd
    - ls
    - echo 'Testing goes here'
  after_script:
    - cd ../
    - pwd

And in the api/asset folder I have a job that’s almost identical, except that it has changes set to:

asset-test
    changes:
      - api/asset/**
      - api/asset/**/**
      - api/asset/.gitlab-ci.yml

Etc.

The issue I’m really having is that when a developer checks in a file to ‘api/asset/*’, sometimse the job webman-test runs, and sometime the job asset-test runs. It seems to depend on the developer which makes no sense.

Changes made exclusively in the api/asset folder resulted in this pipeline executing:

Earlier that day, the exact same branch resulted in these jobs running:

etc. etc.

The job that are executing often have very little to do with what changed, and the jobs appear to be running based on random changes.

We’ve tested this with just isolated files as well, and it’s definitely true that changes:only detection is really just randomly running jobs based on what changed in the various subfolders. We setup test jobs which only echoed to the console to verify the problematic behavior.

The intended behavior is to run the test and build stages on a merge request, and to run the test/build/deploy stages after the merge is complete. Our other jobs which also exhibit this behavior also have this configuration:

  only:
    changes:
      - api/asset/**
      - api/asset/**/**
      - api/asset/.gitlab-ci.yml
    refs:
      - master
      - /^test-.*$/
      - /^release-.*$/

which once again seems to get ignore and seems to fire even when the changes we’re merge in are in other folders.

So… what I really can’t figure out is:

  • how do we get support and is support available (even on a paid plan?)
  • this seemed to work for over 2 months before we got the email letting us know that gitlab.com was being upgraded to the latest version a couple of weeks ago - does anyone know of changes that were made that may impact our jobs that used to work?
  • is Gitlab able to do mono repos like this, or using another technique?
  • if this is a bug, where do I submit it and what would be the priority I could expect?

We are 3 weeks from launch and if I can’t get this working in the next couple of days I need to either split my repos to individual projects or move to a different CI tool in the interim. <- That’s the real decision I’m trying to make in the next few days.

Thanks for any help!
Jason