Tracking dependent artifact version

I’ve a pipeline that consists for 4 stages

  • build
  • build docker image 1 (only for a subset of changes)
  • build docker image 2
  • test
  • deploy_QA
  • deploy_Staging

The artifacts needed in QA are both the build artifacts - docker image 1 and docker image 2.
The docker image 1 however is built only when there are “relevant” changes, managed by “only changes” clause in gitlab yml.

Now, when I have to do a new deploy to QA: I need the artifact versions of both the docker images so that they can be retrieved and deployed appropriately. There are two cases to handle:

  1. Both build jobs are executed, this is an easy case, where the CI_COMMIT_SHORT_SHA or other such variables could be put to use.
  2. For the case where the “build docker image 1” got skipped due to “no changes” and “build docker image 2” executed, how do we handle this ?

I could use “latest” tag as a work around when pushing images to registry, but, that defeats the purpose of “versions”. I won’t know which “version” is deployed and “latest” need not mean the same thing across instances of time. So, I really want to avoid “latest”.

Happy to answer further questions, if any!

gitlab.yml
stages:
- build
- dev:create_infra
- dev:deploy_fluentd
- dev:deploy_app

.before: &before_script
  before_script:
    - apk add --no-cache py3-pip
    - pip3 install awscli
    - pip3 install docker-compose

.plan: &tf_plan
  image: 
    name: hashicorp/terraform:0.12.24
    entrypoint: [""]
  script:
    - echo "Running terraform plan"
    - cd terraform/$MODULE
    - TF_VAR_environment=$DEPLOY_ENV terraform init -input=false -reconfigure
    - TF_VAR_environment=$DEPLOY_ENV terraform plan

.apply: &tf_apply
  image: 
    name: hashicorp/terraform:0.12.24
    entrypoint: [""]
  script:
    - echo "Running terraform apply"
    - cd terraform/$MODULE
    - TF_VAR_environment=$DEPLOY_ENV terraform init -input=false -reconfigure
    - TF_VAR_environment=$DEPLOY_ENV terraform apply -auto-approve # This takes care of deploying the app via AWS constructs

.apply_with_version: &tf_apply_w_version
  image: 
    name: hashicorp/terraform:0.12.24
    entrypoint: [""]
  script:
    - echo "Running terraform apply"
    - cd terraform/$MODULE
    - TF_VAR_environment=$DEPLOY_ENV terraform init -input=false -reconfigure
    - TF_VAR_environment=$DEPLOY_ENV terraform apply -auto-approve -var service_version=$VERSION

build:fluentd:
  stage: build
  image: docker:19.03.1
  services:
    - docker:19.03.1-dind
  variables:
    IMAGE_NAME: "fluentbit"
  only:
    changes:
      - fluentbit/*
  script:
    - apk add --no-cache py3-pip
    - cd fluentbit    
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
    - docker build -f Dockerfile . -t $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA
    - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA
  tags:
    - managed

build:hapee:
  stage: build
  environment:
    name: dev
  image: docker:19.03.1
  services:
    - docker:19.03.1-dind
  variables:
    DOCKER_TLS_CERTDIR: ""
  <<: *before_script    
  only:
    changes:
      - app/*
  script:
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
    - docker-compose build
    - docker tag $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA
    - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA
  tags:
    - managed

create_infra_dev:
  stage: dev:create_infra
  environment:
    name: dev
  variables:
    DEPLOY_ENV: "dev"
    MODULE: "infra"
  tags:
    - managed
  when: on_success
  <<: *tf_apply

deploy_fluentd_dev:
  stage: dev:deploy_fluentd
  environment:
    name: dev
  variables:
    DEPLOY_ENV: "dev"
    MODULE: "fluentbit"
    VERSION: "GIT_SHA" # Work around for now! Would like this to be "resolved" automatically
  tags:
    - managed
  when: on_success
  <<: *tf_apply_w_version

deploy_app_dev:
  stage: dev:deploy_app
  environment:
    name: dev
  variables:
    DEPLOY_ENV: "dev"
    MODULE: "app"
  tags:
    - managed
  when: on_success
  <<: *tf_apply_w_version