Is it possible to run a subset list of jobs / stages?

I would like to be able to designate a list to run a subset of stages when I commit. A full test may take a few hours, so usually when iterating to achieve goals quicker there are only a few tests that are necesary before I would run the full pipe. is this possible?

Currently my closest approximation allows me to run a single task, and set all others to manual. its not terrible, but its not great either. the main blocker for me is I don’t think its possible to use a condition to check if a var is in a list with rules.

Here is an example of the template I use for all jobs. its very hard to read, but the gist of it is that if you define $single_stage to be the job stage you want, then it will run that stage only, and set others to manual.

the logic here for running a single job is on the 2nd condition -

    - if: $single_stage == $CI_JOB_STAGE && $CI_COMMIT_BRANCH != "master" && $CI_COMMIT_BRANCH != "stage"
      when: always
      allow_failure: false

And the last condition, making other tasks manual.

    - if: $single_stage != $CI_JOB_STAGE && $CI_COMMIT_BRANCH != "master" && $CI_COMMIT_BRANCH != "stage"
      when: manual
      allow_failure: true
.job_template_dev: &job_definition_dev
  resource_group: dev_resource
  dependencies: [] # we specify no dependencies so that artifacts dont all get aquired (default), which would orphan the tfstate file.
  rules:
    - if: $all_manual =~ /true/ && $CI_COMMIT_BRANCH != "master" && $CI_COMMIT_BRANCH != "stage"
      when: manual
      allow_failure: false
    - if: $single_stage == $CI_JOB_STAGE && $CI_COMMIT_BRANCH != "master" && $CI_COMMIT_BRANCH != "stage"
      when: always
      allow_failure: false
    - if: $single_stage == 'disabled' && $CI_COMMIT_BRANCH != "master" && $CI_COMMIT_BRANCH != "stage"
      when: always
      allow_failure: false
    - if: $destroy_on_failure == 'true' && $CI_JOB_STAGE == 'cleanup_terraform_destroy_dev' && $CI_COMMIT_BRANCH != "master" && $CI_COMMIT_BRANCH != "stage"
      when: on_failure
      allow_failure: true
    - if: $manual == 'true' && $CI_COMMIT_BRANCH != "master" && $CI_COMMIT_BRANCH != "stage"
      when: manual
      allow_failure: true
    - if: $single_stage != $CI_JOB_STAGE && $CI_COMMIT_BRANCH != "master" && $CI_COMMIT_BRANCH != "stage"
      when: manual
      allow_failure: true
  tags: # the runner is tagged dev
    - dev

This is all not very nice due to the limits of rules and my understanding of them thus far. I’m not sure if being able to define a list with this type of approach would be possible, but it would be nice if it could be done.

Is there a solution to this workflow problem? Ideally it would be great if we could just comment out anything in the stages list, but this produces errors since jobs will reference items that don’t exist in the list.

Being able to do that though would be a decent solution, just commenting out options in the list would be a quick way to run a selection of tests.

stages:

  • sync
  • single_test
  • test_decrypt # ensure decryption of the vault key from the gitlab environment variables can occur
  • pre_vm_vagrant_destroy
  • test_vm
  • init_vm
  • vm_up
  • init_tf_destroy
  • init_tf_delete_state
  • init_vm_config # build a rudimentary vm group initialising promisc mode and other work requirements
  • init_local_deploy # configure the vm group
  • init_local_deploy_destroy
  • deploy_vpn
  • deploy_fsx
  • deploy_softnas_build_ami
  • deploy_softnas
  • deploy_nodes
  • deploy_cloud_workstation
  • deploy_nodes_tf_destroy
  • deploy_no_houdini_vagrant_destroy
  • deploy_nodes_no_nfs_disable_license_server
  • deploy_nodes_vagrant_destroy
  • deploy_end_to_end
  • deploy # test a full cloud deployment in the dev environment, then destroy
  • sleep
  • manual_operations
  • cleanup_tf_destroy
  • manual_cleanup_tf_destroy