Using CI/CD to pull remote terraform modules - deploy tokens?

I’m trying to access resources in another project on the same GitLab instance. In terraform, you reference like so:

module dynamo_terraform_lock_table {
  source = "git::https://git.mycompany.com/infrastructure/resources/tf-modules/terraform-state-lock?ref=0.0.4"

  ...
}

When I try to run a terraform init via CI/CD, the job errors out because it cannot access the target repository.

Executing "step_script" stage of the job script
$ git config --global user.name ctote_test
$ git config --global user.password my-hard-coded-token
$ cd terraform
$ terraform init
Initializing modules...
...
fatal: could not read Username for 'https://git.mycompany.com': No such device or address

I was thinking I could use deploy keys somehow to solve this, but I haven’t been able to figure them out. I created a token called gitlab-deploy-token in both repos and then tried including git config commands to set $CI_DEPLOY_USER and $CI_DEPLOY_PASSWORD but it still gives me errors.

Is there a guide to setting up cross-repo access for CI/CD runners that I’m missing? I think this just boils down to figuring out how I can access resources in another project from the CI/CD pipeline, without hard-coding static credentials.

Hey @ctote,
I stumbled across the problem also months ago. I did this with a technical user (separate user in gitlab with access to the project) and a GitLab Token. I didn’t tried it with a deploy token but this should also work but then you need entering a lot of entries.

The cool part that I found a solution that worked for me. So what I did was to be able in the pipeline doling https clones without entering credentials.

So for this case the git-credential-store comes in. The my script below does the following.

  1. Initialize folder for git-credentials under $HOME/.config/git -> $HOME/.config/git/credentials
  2. Write parsed credentials to $HOME/.config/git/credentials
  3. Enable credential.helper gobally
#!/bin/bash
set -e 
set -o pipefail


if [[ -v "GL_TOKEN" || -v "GITLAB_TOKEN" ]]; then
  if [[ "${CI_PROJECT_URL}" =~ (([^/]*/){3}) ]]; then
    mkdir -p $HOME/.config/git
    echo "${BASH_REMATCH[1]/:\/\//://gitlab-ci-token:${GL_TOKEN:-$GITLAB_TOKEN}@}" > $HOME/.config/git/credentials
    git config --global credential.helper store
  fi
fi

Afterwards you need only to call this very early in your Build Job

terraform:
  stage: prepare
  before_script:
  - init_https_git_clone.sh
  scripts:
  - cd terraform/
  - terraform init

Thank you! I’m not very familiar with this process, could you help break down what’s happening in your example?

The end goal I’d like to get to is having a single gitlab-deploy-token that I can use, or something similar. Does your example do that?

Here’s some more context related to my attempts to use this special deploy token user:

I have a terraform project that references resources in another project. Are there built-in mechanisms for allowing GitLab CICD runners to access multiple projects via a single configuration somewhere? I would love to avoid having to hard-code authentication.

My terraform project resources are in:

- Infrastructure/resources/tf-modules/...

and my live infrastructure is in:

- Infrastructure/live/...

However, when I’m trying to run my live infrastructure, when it attempt to pull resources from the other project:

module dynamo_terraform_lock_table {
  source = "git::https://git.abc.com/infrastructure/resources/tf-modules/dynamodb/terraform-state-lock?ref=0.0.4"
}

My CICD runner errors out:

fatal: could not read Username for 'https://git.abc.com': No such
device or address

Here is the relevant part of my .gitlab-cy.yml:

script:
    - git config --global user.name $CI_DEPLOY_USER
    - git config --global user.password $CI_DEPLOY_PASSWORD
    - cd terraform
    - terraform init
    - terraform plan

The above attempts to use the special deploy token, gitlab-deploy-token but I guess I don’t understand how that’s supposed to work. This seems like the ideal solution since it keeps everything behind the scenes, but I’m not sure what I’m doing wrong.

Below is what I have in my gitlab-ci.yml file for CI/CD. Another work around to the issue of accessing private git repositories with terraform.
What I did was create ssh key for my CI/CD environment and added a known host as variables and this solved my issue. Hope this gives you a head start or a place to build on.|

before_script:

  - rm -rf .terraform

  - terraform --version

  - echo $service_account > ./src/credentials.json

  - eval $(ssh-agent -s)

  - echo "$SSH_KEY" | tr -d '\r' | ssh-add - > /dev/null

  - mkdir -p ~/.ssh && touch ~/.ssh/known_hosts

  - echo "$SSH_KNOWN_HOSTS" >> ~/.ssh/known_hosts