Best approach for running Terraform on docker executors

GitLab CI/CD
, , ,
1

Hi community,

My Gitlab CI environment consists of a runner that executes pipelines on docker+machine executors that spin up VMs on my vmware environment. Each stage in the pipeline executes with a container image (docker:20.10.16).

I am building pipelines to deploy infra with terraform for the first time and my approach is that I have a terraform executable container I am managing with docker compose and this is providing a volume mount from the executor container image for each stage for the project build dir along with an NFS mounted volume for the TF state.

I am happily executing terraform plan and uploading the plan file as an artifact along with the report.

However, when I execute apply it fails because TF was re-initialised before apply since the project build dir won’t have the .terraform dir created from the previous stage’s terraform init. Basically each stage has to do a terraform init since they run on a different container each time and the changes on the project build dir volume do not persist.

So the question is what is the correct approach to this?

My docker-compose.yml:

services:
  terraform:
    image: ${CI_REGISTRY}/${TF_IMAGE}:${TF_IMAGE_TAG}
    volumes:
      - type: volume
        source: tfstate
        target: ${TERRAFORM_TFSTATE_MOUNTPOINT}
        volume:
          nocopy: true
    working_dir: /builds
    environment:
      http_proxy: ${http_proxy}
      https_proxy: ${https_proxy}
      no_proxy: ${no_proxy}
      TF_VAR_vsphere_user: ${TF_VAR_vsphere_user}
      TF_VAR_vsphere_password: ${TF_VAR_vsphere_password}
      TF_VAR_dns_key_secret: ${TF_VAR_dns_key_secret}

volumes:
  tfstate:
    driver_opts:
      type: "nfs"
      o: "addr=${TFSTATE_NFS_SERVER},${TFSTATE_NFS_OPTS}"
      device: ":${TFSTATE_NFS_PATH}"

.gitlab-ci.yml:

stages:
    - terraform_validate
    - terraform_plan
    - terraform_apply

terraform_validate:
    stage: terraform_validate
    image: docker:20.10.16
    variables:
        DOCKER_TLS_CERTDIR: "/certs"
    before_script:
        - |
            set -xv
            http_proxy=$http_proxy
            https_proxy=$https_proxy
            no_proxy=${no_proxy},docker
            apk add --no-cache ca-certificates jq curl
            wget http://ca.anfieldroad.int/certs/anfieldroad-ca-chain-bundle.cert.pem -O /usr/local/share/ca-certificates/anfieldroad-ca-chain-bundle.cert.pem && cat /usr/local/share/ca-certificates/anfieldroad-ca-chain-bundle.cert.pem >> /etc/ssl/certs/ca-certificates.crt  # update-ca-certificates
            mkdir -p /certs/client
            cp /etc/ssl/certs/ca-certificates.crt /certs/client/ca.crt
            alias convert_report="jq -r '([.resource_changes[]?.change.actions?]|flatten)|{\"create\":(map(select(.==\"create\"))|length),\"update\":(map(select(.==\"update\"))|length),\"delete\":(map(select(.==\"delete\"))|length)}'"
            docker info
            docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
    script:
        - |
            set -xv
             echo -en "http_proxy=${http_proxy}\n\
            https_proxy=${https_proxy}\n\
            no_proxy=${no_proxy},docker\n\
            CI_REGISTRY=${CI_REGISTRY}\n\
            TF_IMAGE=${TF_IMAGE}\n\
            TF_IMAGE_TAG=${TF_IMAGE_TAG}\n\
            TF_VAR_vsphere_user=${TF_VAR_vsphere_user}\n\
            TF_VAR_vsphere_password='${TF_VAR_vsphere_password}'\n\
            TF_VAR_dns_key_secret=${TF_VAR_dns_key_secret}\n\
            TERRAFORM_TFSTATE_NFS_HOST=${TERRAFORM_TFSTATE_NFS_HOST}\n\
            TERRAFORM_TFSTATE_NFS_OPTS=${TERRAFORM_TFSTATE_NFS_OPTS}\n\
            TERRAFORM_TFSTATE_NFS_PATH=${TERRAFORM_TFSTATE_NFS_PATH}\n\
            TERRAFORM_TFSTATE_MOUNTPOINT=${TERRAFORM_TFSTATE_MOUNTPOINT}\n" > .env
            echo -en "terraform {\n  backend \"local\" {\n    path = \"${TERRAFORM_TFSTATE_MOUNTPOINT}/terraform.tfstate\"\n  }\n}\n" > tfstate.tf
            docker compose run --rm --volume ${CI_PROJECT_DIR}:${CI_PROJECT_DIR} --workdir ${CI_PROJECT_DIR} terraform init
            docker compose run --rm --volume ${CI_PROJECT_DIR}:${CI_PROJECT_DIR} --workdir ${CI_PROJECT_DIR} terraform validate

terraform_plan:
    stage: terraform_plan
    image: docker:20.10.16
    variables:
        DOCKER_TLS_CERTDIR: "/certs"
        PLAN: "${CI_PROJECT_NAME}_${CI_COMMIT_BRANCH}-${CI_COMMIT_SHORT_SHA}.plan"
        PLAN_JSON: "tfplan.json"
    before_script:
        - |
            set -xv
            http_proxy=$http_proxy
            https_proxy=$https_proxy
            no_proxy=${no_proxy},docker
            apk add --no-cache ca-certificates jq curl
            wget http://ca.anfieldroad.int/certs/anfieldroad-ca-chain-bundle.cert.pem -O /usr/local/share/ca-certificates/anfieldroad-ca-chain-bundle.cert.pem && cat /usr/local/share/ca-certificates/anfieldroad-ca-chain-bundle.cert.pem >> /etc/ssl/certs/ca-certificates.crt  # update-ca-certificates
            mkdir -p /certs/client
            cp /etc/ssl/certs/ca-certificates.crt /certs/client/ca.crt
            alias convert_report="jq -r '([.resource_changes[]?.change.actions?]|flatten)|{\"create\":(map(select(.==\"create\"))|length),\"update\":(map(select(.==\"update\"))|length),\"delete\":(map(select(.==\"delete\"))|length)}'"
            docker info
            docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
    script:
        - |
            set -xv
             echo -en "http_proxy=${http_proxy}\n\
            https_proxy=${https_proxy}\n\
            no_proxy=${no_proxy},docker\n\
            CI_REGISTRY=${CI_REGISTRY}\n\
            TF_IMAGE=${TF_IMAGE}\n\
            TF_IMAGE_TAG=${TF_IMAGE_TAG}\n\
            TF_VAR_vsphere_user=${TF_VAR_vsphere_user}\n\
            TF_VAR_vsphere_password='${TF_VAR_vsphere_password}'\n\
            TF_VAR_dns_key_secret=${TF_VAR_dns_key_secret}\n\
            TERRAFORM_TFSTATE_NFS_HOST=${TERRAFORM_TFSTATE_NFS_HOST}\n\
            TERRAFORM_TFSTATE_NFS_OPTS=${TERRAFORM_TFSTATE_NFS_OPTS}\n\
            TERRAFORM_TFSTATE_NFS_PATH=${TERRAFORM_TFSTATE_NFS_PATH}\n\
            TERRAFORM_TFSTATE_MOUNTPOINT=${TERRAFORM_TFSTATE_MOUNTPOINT}\n" > .env
            echo -en "terraform {\n  backend \"local\" {\n    path = \"${TERRAFORM_TFSTATE_MOUNTPOINT}/terraform.tfstate\"\n  }\n}\n" > tfstate.tf
            docker compose run --rm --volume ${CI_PROJECT_DIR}:${CI_PROJECT_DIR} --workdir ${CI_PROJECT_DIR} terraform init
            docker compose run --rm --volume ${CI_PROJECT_DIR}:${CI_PROJECT_DIR} --workdir ${CI_PROJECT_DIR} terraform plan --out=${CI_PROJECT_DIR}/$PLAN
            docker compose run --rm --volume ${CI_PROJECT_DIR}:${CI_PROJECT_DIR} --workdir ${CI_PROJECT_DIR} terraform show --json $PLAN | convert_report > ${CI_PROJECT_DIR}/${PLAN_JSON}
            cat ${CI_PROJECT_DIR}/${PLAN_JSON}
    artifacts:
        paths:
         - ${CI_PROJECT_DIR}/${PLAN}
        reports:
            terraform: ${CI_PROJECT_DIR}/${PLAN_JSON}
    dependencies:
        - terraform_validate

terraform_apply:
    stage: terraform_apply
    image: docker:20.10.16
    variables:
        DOCKER_TLS_CERTDIR: "/certs"
        PLAN: "${CI_PROJECT_NAME}_${CI_COMMIT_BRANCH}-${CI_COMMIT_SHORT_SHA}.plan"
    before_script:
        - |
            set -xv
            http_proxy=$http_proxy
            https_proxy=$https_proxy
            no_proxy=${no_proxy},docker
            apk add --no-cache ca-certificates jq curl
            wget http://ca.anfieldroad.int/certs/anfieldroad-ca-chain-bundle.cert.pem -O /usr/local/share/ca-certificates/anfieldroad-ca-chain-bundle.cert.pem && cat /usr/local/share/ca-certificates/anfieldroad-ca-chain-bundle.cert.pem >> /etc/ssl/certs/ca-certificates.crt  # update-ca-certificates
            mkdir -p /certs/client
            cp /etc/ssl/certs/ca-certificates.crt /certs/client/ca.crt
            alias convert_report="jq -r '([.resource_changes[]?.change.actions?]|flatten)|{\"create\":(map(select(.==\"create\"))|length),\"update\":(map(select(.==\"update\"))|length),\"delete\":(map(select(.==\"delete\"))|length)}'"
            docker info
            docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
    script:
        - |
            set -xv
             echo -en "http_proxy=${http_proxy}\n\
            https_proxy=${https_proxy}\n\
            no_proxy=${no_proxy},docker\n\
            CI_REGISTRY=${CI_REGISTRY}\n\
            TF_IMAGE=${TF_IMAGE}\n\
            TF_IMAGE_TAG=${TF_IMAGE_TAG}\n\
            TF_VAR_vsphere_user=${TF_VAR_vsphere_user}\n\
            TF_VAR_vsphere_password='${TF_VAR_vsphere_password}'\n\
            TF_VAR_dns_key_secret=${TF_VAR_dns_key_secret}\n\
            TERRAFORM_TFSTATE_NFS_HOST=${TERRAFORM_TFSTATE_NFS_HOST}\n\
            TERRAFORM_TFSTATE_NFS_OPTS=${TERRAFORM_TFSTATE_NFS_OPTS}\n\
            TERRAFORM_TFSTATE_NFS_PATH=${TERRAFORM_TFSTATE_NFS_PATH}\n\
            TERRAFORM_TFSTATE_MOUNTPOINT=${TERRAFORM_TFSTATE_MOUNTPOINT}\n" > .env
            echo -en "terraform {\n  backend \"local\" {\n    path = \"${TERRAFORM_TFSTATE_MOUNTPOINT}/terraform.tfstate\"\n  }\n}\n" > tfstate.tf
            docker compose run --rm --volume ${CI_PROJECT_DIR}:${CI_PROJECT_DIR} --workdir ${CI_PROJECT_DIR} terraform apply ${CI_PROJECT_DIR}/$PLAN
    dependencies:
        - terraform_plan
    when: manual
    only:
        - master

The result:

Error: Inconsistent dependency lock file
│ 
│ The following dependency selections recorded in the lock file are
│ inconsistent with the configuration in the saved plan:
│   - provider registry.terraform.io/hashicorp/dns: required by this configuration but no version is selected
│   - provider registry.terraform.io/hashicorp/vsphere: required by this configuration but no version is selected
│ 
│ A saved plan can be applied only to the same configuration it was created
│ from. Create a new plan from the updated configuration.
╵
â•·
│ Error: Inconsistent dependency lock file
│ 
│ The given plan file was created with a different set of external dependency
│ selections than the current configuration. A saved plan can be applied only
│ to the same configuration it was created from.
│ 
│ Create a new plan from the updated configuration.
╵
Cleaning up project directory and file based variables
00:00
ERROR: Job failed: exit code 1

How can I get the .terraform directory init to persist throughout the pipeline given:

  1. Each stage executes on a new container
  2. The plan and apply stages of the pipeline likely won’t execute on the same executor container since there is a review and approve stage in between where the plan output will be checked, approval and merge to master, only then will apply be executed by pressing the “play” button on the pipeline to execute apply manually. It’s likely by then the docker machine VM has idled out and been destroyed by this point.

My initial thought is that at the beginning of the pipeline a stage executes init and then the directory is zipped and uploaded as an artifact.

Then this artifact will be downloaded and unzipped for plan and apply stages.

Any thoughts please?

2

I have resolved this issue myself, I realised that setting dependencies copied artifacts and made them available to the stages depending on them.

As such I created a first “terraform_init_validate” stage and made that a dependancy of the plan and apply stages.

I would still like to hear from anyone any other things to consider in such a pipeline though please :slight_smile:

1 Like