GitLab CI Terraform APPLY using plan requires authentication

GitLab CI/CD
,
1

This is the .gitlab-ci.yml I have been using:

deploy-infra-prd:
  stage: deploy-infra-prd
  timeout: 1h 30m
  <<: *before_script
  script:
    - rm -rf .terraform
    - terraform init -backend-config=address=${PROJECT_ADDRESS} 
        -backend-config=lock_address=${PROJECT_ADDRESS}/lock 
        -backend-config=unlock_address=${PROJECT_ADDRESS}/lock 
        -backend-config=lock_method=POST 
        -backend-config=unlock_method=DELETE 
        -backend-config=retry_wait_min=5

      terraform apply --auto-approve out.${ENVIRONMENT}.plan

  resource_group: lock_terraform
  variables:
    ENVIRONMENT: prd
    PROJECT_ADDRESS: $CI_API_V4_URL/projects/$CI_PROJECT_ID/terraform/state/infra
    TF_HTTP_USERNAME: gitlab-ci-token
    TF_HTTP_PASSWORD: ${CI_JOB_TOKEN}

During init, notice there’s we do not use -backend-config=password=${CI_JOB_TOKEN} so that TF_HTTP_PASSWORD can be used and we can see the below in Terraform debug logs:

2024-01-05T02:58:51.034Z [DEBUG] GET https://gitlab.com/api/v4/projects/<my-gitlab-project-id>/terraform/state/infra

[51](https://gitlab.com/<my-gitlab-repos>/-/jobs/5865587700#L51)Initializing modules...
...
Terraform has been successfully initialized!

The init seems to work well and is taking the 2 ENV variables:

TF_HTTP_USERNAME: gitlab-ci-token
TF_HTTP_PASSWORD: ${CI_JOB_TOKEN}

However when it comes to APPLY the plan file, we have got the below:

2024-01-05T02:59:14.254Z [INFO]  CLI args: []string{"terraform", "apply", "--auto-approve", "out.prd.plan"}
2024-01-05T02:59:14.254Z [DEBUG] Attempting to open CLI config file: /root/.terraformrc
2024-01-05T02:59:14.254Z [DEBUG] File doesn't exist, but doesn't need to. Ignoring.
2024-01-05T02:59:14.255Z [DEBUG] ignoring non-existing provider search directory terraform.d/plugins
2024-01-05T02:59:14.255Z [DEBUG] ignoring non-existing provider search directory /root/.terraform.d/plugins
2024-01-05T02:59:14.255Z [DEBUG] ignoring non-existing provider search directory /root/.local/share/terraform/plugins
2024-01-05T02:59:14.255Z [DEBUG] ignoring non-existing provider search directory /usr/local/share/terraform/plugins
2024-01-05T02:59:14.255Z [DEBUG] ignoring non-existing provider search directory /usr/share/terraform/plugins
2024-01-05T02:59:14.257Z [INFO]  CLI command args: []string{"apply", "--auto-approve", "out.prd.plan"}
2024-01-05T02:59:16.408Z [DEBUG] checking for provisioner in "."
2024-01-05T02:59:16.408Z [DEBUG] checking for provisioner in "/usr/local/bin"
2024-01-05T02:59:16.413Z [INFO]  backend/local: starting Apply operation
2024-01-05T02:59:16.413Z [DEBUG] POST https://gitlab.com/api/v4/projects/<my-gitlab-project-id>/terraform/state/infra/lock
╷
│ Error: Error acquiring the state lock
│ 
│ Error message: HTTP remote state endpoint requires auth

We noticed that if we use APPLY without a plan after an init, it will work, but using a plan file we are having the above issues.

I have been stuck with this migrating our state files from S3 bukcet to Gitlab managed state files.

What could have been missing?

2

After a bit more searching, found this Terraform state lock in CI "endpoint requires auth" for plan (#338482) · Issues · GitLab.org / GitLab · GitLab

Turn out it’s when I create my plan, I was using:

      -backend-config=username=gitlab-ci-token 
      -backend-config=password=${CI_JOB_TOKEN}

Since the token is a short live token, it expired when we apply the plan.

Crisis averted. We can now happily migrate to Gitlab managed state files.