GitLab Terraform AWS Pipeline issue

Problem to solve

I am trying to setup simple CICD pipeline in Gitlab to deploy resources in AWS using Terraform IAC. All Terraform commands are working fine if running locally on laptop and able to deploy all resources to AWS as expected. However, same setup if I try to run it through GitLab is not working.

Our requirement: We have separate GitLab instances for Dev and Prod. I have separate folder under env directory for prod and dev respectively where I am running terraform commands. So we want to configure pipeline to deploy terraform IAC with condition if $CI_SERVER_URL is prod URL then run got to env/prod directory and run terraform command else go to env/dev directory and run terraform command.

Gitlab directory is stored in root directory of the project i.e. .project

Steps to reproduce

I am sharing my .gitlab-ci.yml file and output I am getting in Gitlab logs through DBUG_MODE.

Configuration

Folder Structure

Project

├── README.md
├── .gitlab-ci.yml.md
├── .gitignore
├── __init__.py
├── env
│   ├── dev
│   │   ├── main.tf
│   │   ├── outputs.tf
│   │   ├── terraform.tfvars
│   │   ├── tfplan.json
│   │   └── variables.tf
│   ├── prod
│   │   ├── main.tf
│   │   ├── outputs.tf
│   │   ├── terraform.tfvars
│   │   ├── tfplan.json
│   │   └── variables.tf
│   └── stage
│       ├── main.tf
│       ├── outputs.tf
│       ├── terraform.tfvars
│       ├── tfplan.json
│       └── variables.tf
└── modules
    ├── __init__.py
    ├── api_gateway
    │   ├── main.tf
    │   ├── outputs.tf
    │   └── variables.tf
    ├── dynamodb
    │   ├── main.tf
    │   ├── outputs.tf
    │   └── variables.tf

GitLab Configuration File

variables:
  AWS_ACCESS_KEY_ID: ${AWS_ACCESS_KEY_ID}
  AWS_SECRET_ACCESS_KEY : ${AWS_SECRET_ACCESS_KEY}
  AWS_DEFAULT_REGION: ${AWS_DEFAULT_REGION}
  # CI_DEBUG_TRACE: "true"

image:
  name: registry.gitlab.com/gitlab-org/gitlab-build-images:terraform
  entrypoint:
    - '/usr/bin/env'
    - 'PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID} AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY} AWS_DEFAULT_REGION=${AWS_DEFAULT_REGION}’'



stages:
  - .pre
  - validate
  - plan
  - apply

# Define the job to set the environment folder in the init stage
set_env_folder:
  stage: .pre
  script:
    - echo "starting setting CI_ENV_FOLDER"
    - pwd
    - |
      if [ "$CI_SERVER_URL" == "https://gitlab.prod.com" ]; then
        echo "Running on production GitLab"
        export CI_ENV_FOLDER="env/prod"
      else
        echo "Running on development GitLab"
        export CI_ENV_FOLDER="env/dev"
      fi
      echo "Using CI_ENV_FOLDER=$CI_ENV_FOLDER" > .env
  artifacts:
    reports:
      dotenv: .env


cache:
  paths:
    - $CI_ENV_FOLDER/.terraform

before_script:
  - echo "Sourcing .env file:"
  - cat .env
  - source .env
  - echo "The secret API key is $CI_ENV_FOLDER"
  - pwd
  - cd $CI_ENV_FOLDER
  - pwd
  - terraform --version
  - terraform init


validate:
  stage: validate
  script:
    - echo "Inside plan"
    - pwd
    - terraform validate

plan:
  stage: plan
  script:
    - echo "Inside plan"
    - pwd
    - terraform plan -out="plan.tfplan"
  dependencies:
    - validate
  artifacts:
    paths:
      - $CI_ENV_FOLDER/plan.tfplan

apply:
  stage: apply
  script:
    - terraform apply "$CI_ENV_FOLDER/plan.tfplan"
  dependencies:
    - plan
  when: manual

GitLab Output:

+ export 'RUNNER_TEMP_PROJECT_DIR=/builds/service.tmp'
+ cd /builds/service
+ echo '$ echo "Sourcing .env file:"'
$ echo "Sourcing .env file:"
+ echo 'Sourcing .env file:'
Sourcing .env file:
+ echo '$ cat .env'
$ cat .env
+ cat .env
cat: can't open '.env': No such file or directory
+ runner_script_trap
+ exit_code=1
+ out_json='{"command_exit_code": 1, "script": "/scripts-1234-1234/step_script"}'
+ echo 
+ echo '{"command_exit_code": 1, "script": "/scripts-1234-1234/step_script"}'
+ exit 0
Uploading artifacts for failed job
.
.
.
.
++ /usr/bin/gitlab-runner-helper --version
++ echo 'Uploading artifacts...'
Uploading artifacts...
++ /usr/bin/gitlab-runner-helper artifacts-uploader --url https://gitlab.com/ --token [MASKED] --id 1234 --path .env --name .env --artifact-format gzip --artifact-type dotenv
WARNING: .env: no matching files. Ensure that the artifact path is relative to the working directory (/builds/service) 
ERROR: No files to upload                          
+ exit 0
+ runner_script_trap
+ exit_code=0
+ out_json='{"command_exit_code": 0, "script": "/scripts-1234-1234/upload_artifacts_on_failure"}'
+ echo ''
+ echo '{"command_exit_code": 0, "script": "/scripts-1234-1234/upload_artifacts_on_failure"}'
+ exit 0
Cleaning up project directory and file based variables

Versions

Please select whether options apply, and add the version information.

• Self-managed