CI/CD Pipeline - remote include files that include themselves other local files (on remote project)

GitLab CI/CD
,
1

Is it possible to include remote files that include themselves other files?

Hi there,

I have 2 projects:

  • Project automation-templates has all the defined pipelines and jobs
automation-templates
├── .gitlab-ci
│   ├── jobs
│   │   ├── check.yml     # includes the child pipelines in dir "pipelines"
│   │   ├── deploy.yml
│   ├── pipelines
│   │   ├── check.yml     # includes the template jobs in dir  "templates"
│   │   ├── common.yml
│   │   ├── deploy.yml
│   ├── scripts
│   │   ├── apply.sh
│   │   ├── echo.sh
│   ├── templates
│   │   ├── check.yml    # uses the scripts in dir "scripts"
│   │   └── deploy.yml
│   └── variables.yml
├── .gitlab-ci.yml

Sample content of the root automation-templates/.gitlab-ci.yml

stages:
  - check
  - deploy

workflow:
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH 
    
include:
 - ".gitlab-ci/variables.yml"
 - ".gitlab-ci/jobs/*.yml"

Sample content of the file automation-templates/.gitlab-ci/jobs/check.yml

check:
  stage: check
  trigger:
    include: 
      - local: "/.gitlab-ci/pipelines/check.yml"
    strategy: depend
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

  • Project my-app where I would like to use the pipeline declared in automation-templates (see above) by having this following my-app/.gitlab-ci.yml 
include:
  - project: automation-templates
    ref: main
    file: .gitlab-ci.yml

This is working for the 1rst level of inclusion. Indeed I have my pipeline triggered in my-app as the my-app/.gitlab-ci.yml includes the remote automation-templates/.gitlab-ci.yml

But it does work at the level of the included files /.gitlab-ci/jobs/*.yml (on remote project automation-templates because I got this error on my-app pipeline

Found errors in your .gitlab-ci.yml:
Local file `/.gitlab-ci/pipelines/check-talos.yml` does not exist!

I tried many configurations like indicated in the online documentation without success.

So I would like to know if such kind of use case should work in GitLab? This is a possible feature and I misconfigured something or it could be a bug in a “corner case”?

Thanks for your help.

2

Hi,

Did you get any solution for this issue ?
I have similar kind of implementation.

Thanks

3

I’ve tried this and the only way I got it to work was by including local files as if they were remote files.

include:
  - project: 'automation-templates'
    ref: 'main'
    file:
      - ".gitlab-ci/variables.yml"
      - ".gitlab-ci/jobs/*.yml"

works fine until you need to make a branch, at which point you have to change the ref id to test it but then change it to the target branch before merge. or you could do something like

include:
  - project: 'automation-templates'
    # commit_branch will be empty for mrs, and source branch will be empty if not mr
    ref: '$CI_COMMIT_BRANCH$CI_MERGE_REQUEST_SOURCE_BRANCH_NAME'

but then you have to make sure to make a branch of your templates to match the branch of every task you work.
You might be able to conditionally change the include with rules, but that starts down a different nightmare.
Final option is to use a variable… but it has to be a variable thats defined before the pipeline is setup, like a repository variable. so you would have a variable for project_a_template_ref that would point to a ref in the template project. Means you have to do some management of what the var is assigned to especially if testing new templates, but possibly the cleanest solution.

4

New option I just found, “spec::inputs” which is added to the header section of a yaml file (the area above a “—”)
so in each template that has an include use

spec:
  inputs:
    branch:
      default: main
---
include:
  - project: 'automation-templates'
    ref:  $[[ inputs.branch ]]
    files: ...

then when you import in another repo use

include:
  - project: automation-templates
    ref: main
    inputs:
      branch: main
    file: .gitlab-ci.yml