Difficulty running stages from included configs

We currently store our CI configs in a separate project and use the settings in the UI to link them.
For various reasons we want to move to using a .gitlab-ci.yml in the repo itself that includes these files from the cicd repo.

My current attempt is (note that the existing company-backend/.gitlab-ci.yml works perfectly):

include:
  - project: minigrid/cicd
    ref: 'chore/run-quality-separately'
    file:
      - 'company-backend/.gitlab-ci.yml'

# Added this after error described below - these are defined in the included config
stages:
  - TestAndBuild
  - AdditionalLinting
  - Deploy
  - Validate
  - ApproveQA
  - Live
  - PostJobs

Putting this into CI Lint is giving the following error:

image1896×1618 176 KB

I added the stages explicitly (as above) - which I wasn’t expecting to need to do since they are defined in the linked file - but still no luck.
My understanding from the docs is that the entire linked file is essentially merged with the .gitlab-ci.yml to create the yaml that runs. What am I missing?

I’ve omitted the included company-backend/.gitlab-ci.yml file since a) it works already and has been unchanged b) is 350+ lines

Hi @phil122

Have you run this pipeline or just put it through the lint?

No I haven’t yet - I can’t figure out how to run this pipeline in such a way that it overrides the project settings (which I cannot change as it will disrupt my team). If you know of a way then I can definitely give it a whirl!

I was running it in the CI lint tool with “Simulate” checked - which I think should perform the inclusion

Interesting!

Looking at the docs for include I wonder whether you need a leading slash on /company-backend/.gitlab-ci.yml?

Also, is chore/run-quality-separately a tag or branch?

The leading slash doesn’t seem to make any difference.

chore/run-quality-separately is a branch. If I change it to something deliberately wrong then CI Lint recognises the missing ref.

Interestingly, pasting the included file into CI Lint also fails. So I suspect there are further configuration settings that are currently being provided by the project settings that I am failing to provide…

Interesting! So it sounds like this isn’t anything obvious, at least AFAICT.

I wonder what happens if you make a simple branch on that repo, like testci and remove everything in the .gitlab-ci.yml except a really simple job, like:

stages:
    - test

test:
    script:
        - echo "Hello, world!"

Hopefully that will work fine, then I’d change the branch name from testci to chore/testci and see if that makes a difference.

If all of that works fine, then I’d guess it’s something in your .gitlab-ci.yml that you are including. It’s hard to know what that would be, but you could try linting that file in its original minigrid/cicd project, and cutting out the stages and jobs that you aren’t using in the downstream project, and see if that lints OK.

Thanks - the test job works perfectly - good idea.

I’ll coordinate with the devops team to find what the problem inside the cicd/.gitlab-ci.yml might be

OK took me a long time to get back on this but I think I’ve debugged why I’m getting this error.

I think it’s because the anchors with rules in the included file all resolve to false, which means none of the jobs would run.

Take this minimal example:

.automatically_builds: &automatically_builds
  rules:
    - if: '"false" == "merge_request_event"'

image: docker:${Docker_Image_Jobs_Version}

services:
  - docker:${Docker_Image_Jobs_services_Version}

variables:
  DOCKER_DRIVER: overlay2

stages:
  - TestAndBuild

build-push:
  <<: *automatically_builds
  stage: TestAndBuild
  script:
    - echo "1"

If I remove <<: *automatically_builds or make it always resolve to true then it works fine. But with it I get the No stages / jobs for this pipeline. error.
Note that pasting this directly into CI Lint works fine, its only when included that it fails.

I’m not sure if this should be considered a bug with CI Lint or if I’m expecting too much. Maybe the file is really supposed to be considered invalid in these cases ¯_(ツ)_/¯

Anyway just thought I’d report back

Interesting! Should this:

be:

.automatically_builds: &automatically_builds
  rules:
    - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'

maybe?

Yeah sorry that is what it is in the real file, and if I change to that in the minimal example it fails also.
I just changed it to the contrived example to test.
It passes if I update to - if: ' "merge_request_event" == "merge_request_event"'

I wonder whether you have a workflow section, like this:

workflow:
    rules:
        - if: $CI_MERGE_REQUEST_ID
          when: never
        - if: $CI_COMMIT_TAG
          when: never
        - when: always

?

No, we’re not

Also as stated above it fails CI Lint when including just:

.automatically_builds: &automatically_builds
  rules:
    - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'

image: docker:${Docker_Image_Jobs_Version}

services:
  - docker:${Docker_Image_Jobs_services_Version}

variables:
  DOCKER_DRIVER: overlay2

stages:
  - TestAndBuild

build-push:
  <<: *automatically_builds
  stage: TestAndBuild
  script:
    - echo "1"

Perhaps I’m misunderstanding your question a

Interesting! Your fragment there passes the lint for me.

The reason for my question is that GitLab has “MR” pipelines and “branch” pipelines. You only want the MR pipelines, I think, so you probably want to say something like:

workflow:
    rules:
        - if: $CI_MERGE_REQUEST_ID
          when: always
        - when: never

(so, the opposite of my config).

Just a quick follow-up to this. If I ask the lint to Simulate a pipeline created for the default branch the config does fail, because there is nothing in that fragment that will cause the build-push job to run on the default branch. It will pass if you change it to this:

.automatically_builds: &automatically_builds
  rules:
    - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
      when: never
    - when: always

image: docker:${Docker_Image_Jobs_Version}

services:
  - docker:${Docker_Image_Jobs_services_Version}

variables:
  DOCKER_DRIVER: overlay2

stages:
  - TestAndBuild

build-push:
  <<: *automatically_builds
  stage: TestAndBuild
  script:
    - echo "1"

So, either you really are missing some when clauses, or perhaps the problem is in a part of the file you haven’t posted here?