Dynamic input options with separated gitlab-ci files

GitLab CI/CD
1

Problem to solve

Greetings. I used Gitlab CI as my main CI/CD solutions. Since few days I tried to implement input variables with options to select a target VM where my services are deployed. I try to implement this to be more flexible.

All of my IaC are done as bellow :

  • A .gitlab-ci.yml file which is our entrypoint with only few statements :
    • Includes statements per branch to include a gitlab-ci.yml file for each environment (e.g. .ci/.gitlab-ci.yml.
    • A dummy job to avoid a known Gitlab limitation (during Merge request the verification failed because there is no job detected inside .gitlab-ci.yml, so we hardcoded a dummy job)
    • variables statements with a variable named VM_SELECTION_NAME which contain options (list of VM)

I try to find a way to make my options list dynamic. For example, I want a certain list for my develop branch, another for staging and finally an another one for the production environment.

I make some searches but it seem that it’s impossible to declare the options inside the child files because GitLab construct the form from the .gitlab-ci.yml and the child files are imported only after created the pipeline.

The only solution that I found was to list all VM from all environments inside the same variable. Nonetheless, I don’t like this solution because in case of a large amont of VM it makes the list infinite.

I also search a way do implement a logic inside the options but Gitlab doesn’t support Logic in this statement.

Steps to reproduce

To reproduce you need three files and two branches:

  • An entrypoint .gitlab-ci.yml
  • .ci folder which contain following files
    • A first file named <what_you_want>.gitlab-ci.yml
    • A second named also <what_you_want>.gitlab-ci.yml
# Entrypoint Gitlab CI file

include:
  - local: ".ci/development.gitlab-ci.yml"
    rules:
      - if: '$CI_COMMIT_BRANCH == "develop" && $CI_PIPELINE_SOURCE == "web"'
        when: always
  - local: ".ci/staging.gitlab-ci.yml"
    rules:
      - if: '$CI_COMMIT_BRANCH == "staging" && $CI_PIPELINE_SOURCE == "web"'
        when: always
  - local: ".ci/production.gitlab-ci.yml"
    rules:
      - if: '$CI_COMMIT_BRANCH == "master" && $CI_PIPELINE_SOURCE == "web"'
        when: always

dummy_job:
  stage: .pre
  rules:
    - when: never
  script:
    - echo 'To avoid Gitlab limitation'

# First child file

stages:
  - scan
  - deploy

variables:
  VM_SELECTION_NAME:
    value: 'VM1'
    options:
      - 'VM1'
      - 'VM2'
    description: 'Select a VM'

include:
  - project: secret
    ref: my_scan_job@18.9.6
    file: 'secret'
  - project: secret
    ref: my_deploy_job@18.9.6
    file: secret

my_scan_job:
  stage: scan
  environment: Development
  variables:
    DOCKER_COMPOSE_FILE: 'docker-compose.yml --env-file=.env'

my_deploy_job:
  extends: my_deploy_job
  stage: deploy
  environment: Development
  variables:
    ENV_FILE: .env
    STACK_NAME: my-stack
    STACK_FILE: docker-compose.yml
    PORTAINER_API: my-portainer-api
    PORTAINER_TOKEN: my-portainer-token
  rules:
    - if: '$VM_SELECTION_NAME == "VM1"'
      when: manual
      variables:
        PORTAINER_SERVER_ID: $VM1_SERVER_ID
    - if: '$VM_SELECTION_NAME == "VM2"'
      when: manual
      variables:
        PORTAINER_SERVER_ID: $VM2_SERVER_ID

# Second child file

stages:
  - scan
  - deploy

variables:
  VM_SELECTION_NAME:
    value: 'STAGING-VM1'
    options:
      - 'STAGING-VM1'
      - 'STAGING-VM2'
    description: 'Select a VM'

include:
  - project: secret
    ref: my_scan_job@18.9.6
    file: 'secret'
  - project: secret
    ref: my_deploy_job@18.9.6
    file: secret

my_scan_job:
  stage: scan
  environment: Staging
  variables:
    DOCKER_COMPOSE_FILE: 'docker-compose.yml --env-file=.env'

my_deploy_job:
  extends: my_deploy_job
  stage: deploy
  environment: Staging
  variables:
    ENV_FILE: .env
    STACK_NAME: my-stack
    STACK_FILE: docker-compose.yml
    PORTAINER_API: my-portainer-api
    PORTAINER_TOKEN: my-portainer-token
  rules:
    - if: '$VM_SELECTION_NAME == "STAGING-VM1"'
      when: manual
      variables:
        PORTAINER_SERVER_ID: $STAGING_VM1_SERVER_ID
    - if: '$VM_SELECTION_NAME == "STAGING-VM2"'
      when: manual
      variables:
        PORTAINER_SERVER_ID: $STAGING_VM2_SERVER_ID

If you do so you will see that Gitlab don’t detect

Configuration

I don’t have a specific confguration.

Versions

I’m currently using Self-Managed instance in 18.5.1 Community Edition and I have 3 runners :

  • Runner 1 : 17.6.1 => linux/amd64

  • Runner 2 : 17.6.1 => linux/amd64

  • Runner 3 : 17.7.0 => linux/amd64

:warning: Our Runners will be update today or tommorow to the last version 18.5.0.

Helpful resources

I only found few resources, I mainly used Gitlab inputs documentation and make some search on web but I don’t find any similar case.