Gitlab as Terraform http backend failing authentication

I’ve been working to get a monorepo for Terraform up and running. Terraform is executed inside a container. However, when I try to init the backend for http pointing towards my self-hosted Gitlab, it comes back as requiring auth:

Successfully configured the backend "http"! Terraform will automatically
use this backend unless the backend configuration changes.
2021-04-15T13:02:00.815Z [TRACE] Meta.Backend: instantiated backend of type *http.Backend
2021-04-15T13:02:00.815Z [DEBUG] checking for provisioner in "."
2021-04-15T13:02:00.815Z [DEBUG] checking for provisioner in "/bin"
2021-04-15T13:02:00.815Z [INFO]  Failed to read plugin lock file .terraform/plugins/linux_amd64/lock.json: open .terraform/plugins/linux_amd64/lock.json: no such file or directory
2021-04-15T13:02:00.815Z [TRACE] Meta.Backend: backend *http.Backend does not support operations, so wrapping it in a local backend
2021/04/15 13:02:00 [DEBUG] GET https://gitlab.my_domain.tld/api/v4/projects/2/terraform/state/infrastructure
Error refreshing state: HTTP remote state endpoint requires auth

I’ve confirmed that my project token does work by using curl against the project with the token that has API access:

bash-5.1# curl -k -I "https://gitlab.my_domain.tld/api/v4/projects?<my_token_id>"
HTTP/2 200 
server: nginx
date: Wed, 14 Apr 2021 23:59:43 GMT
content-type: application/json
vary: Accept-Encoding
cache-control: no-cache
link: <https://gitlab.my_domain.tld/api/v4/projects?<my_token_id>=&membership=false&order_by=created_at&owned=false&page=1&per_page=20&repository_checksum_failed=false&simple=false&sort=desc&starred=false&statistics=false&wiki_checksum_failed=false&with_custom_attributes=false&with_issues_enabled=false&with_merge_requests_enabled=false>; rel="first", <https://gitlab.my_domain.tld/api/v4/projects?<my_project_token>=&membership=false&order_by=created_at&owned=false&page=1&per_page=20&repository_checksum_failed=false&simple=false&sort=desc&starred=false&statistics=false&wiki_checksum_failed=false&with_custom_attributes=false&with_issues_enabled=false&with_merge_requests_enabled=false>; rel="last"
vary: Origin
x-content-type-options: nosniff
x-frame-options: SAMEORIGIN
x-gitlab-feature-category: projects
x-next-page:
x-page: 1
x-per-page: 20
x-prev-page:
x-request-id: 01F39D73G8ZXPJ1E1ZAK0ZS860
x-runtime: 0.184197
x-total: 1
x-total-pages: 1
strict-transport-security: max-age=31536000
referrer-policy: strict-origin-when-cross-origin

bash-5.1# 

The credentials aren’t passed via -backend-config nor are they put directly in the backend.tfvars file I have with the backend:

backend.tf:

terraform {
    backend "http" { }
}

backend.tfvars:

lock_method = "POST" 
unlock_method = "DELETE" 
retry_max = "3" 
skip_cert_verification = true

And I’ve confirmed with an export that the values for TF_HTTP_USERNAME and TF_HTTP_PASSWORD are being set in the environment variables. Hell, I even set them as global variables:

declare -x TERRAFORM_VERSION="0.15.0"
declare -x TF_HTTP_PASSWORD="[MASKED]"
declare -x TF_HTTP_USERNAME="project_2_bot"
declare -x TF_LOG="trace"
declare -x TF_PASSWORD="[MASKED]"
declare -x TF_USERNAME="project_2_bot"
declare -x bot_2_token="[MASKED]"

I checked the Gitlab logs for api_json.json, and I see requests coming in, but with INFO they don’t tell me much, just that an attempt came in:

{
  "time": "2021-04-15T13:02:00.924Z",
  "severity": "INFO",
  "duration_s": 0.00915,
  "db_duration_s": 0.00207,
  "view_duration_s": 0.00708,
  "status": 401,
  "method": "GET",
  "path": "/api/v4/projects/2/terraform/state/infrastructure",
  "params": [],
  "host": "gitlab.my_domain.tld",
  "remote_ip": "10.4.6.95, 127.0.0.1",
  "ua": "Go-http-client/1.1",
  "route": "/api/:version/projects/:id/terraform/state/:name",
  "queue_duration_s": 0.011228,
  "db_count": 2,
  "db_write_count": 0,
  "db_cached_count": 1,
  "cpu_s": 0.017418,
  "mem_objects": 10346,
  "mem_bytes": 423664,
  "mem_mallocs": 1771,
  "correlation_id": "01F3ASZH67FMVS0449NC289C47",
  "meta.caller_id": "/api/:version/projects/:id/terraform/state/:name",
  "meta.remote_ip": "10.4.6.95",
  "meta.feature_category": "infrastructure_as_code",
  "meta.client_id": "ip/10.4.6.95"
}

I’m trying to find out how to increase the log level to capture all events coming in for the API, but so far nothing.

Every single time I attempt to run a command against the backend, it comes back as:

Error refreshing state: HTTP remote state endpoint requires auth

I’ve tried with Terraform version 0.14.x (can’t remember what the latest was) and 0.15

And I’m running out of avenues to look. Anyone experience this or give ideas on how to troubleshoot it further?

Gitlab Self Hosted, running GitLab Enterprise Edition 13.10.0-ee

Hi @MichaelSweikata

You can also use native support for Terraform using Infrastructure as code with Terraform and GitLab | GitLab

If you want/have to use custom solution:
I don’t see TF_HTTP_ADDRESS variable set.
These are the available env variables Backend Type: http - Terraform by HashiCorp
And this is how you would set backend to GitLab using CLI options

PROJECT_ID="<gitlab-project-id>"
TF_USERNAME="<gitlab-username>"
TF_PASSWORD="<gitlab-personal-access-token>"
TF_ADDRESS="https://gitlab.com/api/v4/projects/${PROJECT_ID}/terraform/state/old-state-name"

terraform init \
  -backend-config=address=${TF_ADDRESS} \
  -backend-config=lock_address=${TF_ADDRESS}/lock \
  -backend-config=unlock_address=${TF_ADDRESS}/lock \
  -backend-config=username=${TF_USERNAME} \
  -backend-config=password=${TF_PASSWORD} \
  -backend-config=lock_method=POST \
  -backend-config=unlock_method=DELETE \
  -backend-config=retry_wait_min=5

If you setup the env variables to match the above CLI command it should work.

Sorry, neglected to include my CI entry that does the request:

- terraform -chdir=./app_teams/${team_folder} init -reconfigure -backend-config="address=https://gitlab.my_domain.tld/api/v4/projects/${CI_PROJECT_ID}/terraform/state/${team_folder}" -backend-config="lock_address=https://gitlab.my_domain.tld/api/v4/projects/${CI_PROJECT_ID}/terraform/state/${team_folder}/lock" -backend-config="unlock_address=https://gitlab.my_domain.tld/api/v4/projects/${CI_PROJECT_ID}/terraform/state/${team_folder}/lock"

I don’t think your variables in backend.tfvars are applied to backend at all. You do not reference them in backend.tf at all.

I have never used variables in backend config before, but I think it should look like:

data "terraform_remote_state" "foo" {
  backend = "http"
  config = {
    lock_method = var.lock_method
    unlock_method = var.unlock_method
    retry_max = var.retry_max
    skip_cert_verification = var.skip_cert_verification
  }
}

and you must have related variable definition for each variable as well.

Or just set everything using env variables.

But this isn’t for a data lookup, it should just be for the initialization. Unless you need to have the data for remote state for initialization?

Make sure the token you are using has api scope.
Remove any existing .terraform directory from project root

Also just my opinion, but specifying options on 3 places (env variables, cli options and backend.tf) isn’t really easy to troubleshoot, but you might have your reasons.

I don’t plan to do all three locations, I’m just trying anything to help make it work. Ideally I want the base http backend, and pass everything via the init cli.

And confirmed that the project token has api access.

All right. So you don’t have .terraform folder in your project root (it is known that any existing config could mess up backend init - alternative way is to use -reconfigure), that would be in ./app_teams/${team_folder} and have you tried to initialize it only from cli only using -backend-config and no env variables?
According to terraform source that message is returned on http.StatusUnauthorized which would be 401.
You can also check nginx logs (or other reverse proxy if you are not using the bundled one) for requests coming in. You can also alter nginx config to dump the entire HTTP request. I think it is easier than making GitLab dump it.
And there is also this issue, but you don’t have . in username, maybe same bug applies to password? Just guessing.

EDIT: looking at this PR password is not URL encoded as well at the moment. This might be the root cause if your token contains some URL specific characters

Okay, sorry, had a life issue yesterday, started back up today.

So I think I found the cause.

I started fresh today, pulled a new container, pulled down my code, switched to the test branch.

I started the process scientifically. First, fresh everything, no exports in the shell, and the use of the same token.

Test #1 was everything in tfvars, and the backend file was blank (except for the declaration of the backend). Same problem, http auth 401.
Test #2 was everything in the backend file, no tfvars. Username, password, all of it. Same problem, http auth 401.

So then I paused, evaluated whether or not it could be the token. So, I created a new project token, project_2_bot1, keeping the old one as a just in case. Gave it the same permissions (api), and tested it in the same ways. Both, 401.

And then I had a moment. I created a new token, gave it everything, api, read/write repository, etc., and tested that. 200 code.

I understand the gitlab docs here:

4. Create a Personal Access Token with the api scope.

I think that’s incorrect. I know I’m using a project token instead of a personal token, but I think the permissions need to be more.

Thank you Michael!!

From now on I wont trust documentation fully… I wasted about 2 hours troubleshooting this and all I needed was a Project Token and in the documentation it does say Personal Access Token.

i tried both Personal Access Token and then project Access Token but both never worked for me
i thing that it depends on the situation
btw i’am not storing my backend on gitlab CI

There’s a lot of things it can depend on, but if you’re having the same issue, I would set up the same logging I did (https://www.terraform.io/docs/internals/debugging.html) and read the actual output of what’s occurring from Terraform’s perspective. If it’s a 401, it’s the token, or other authentication issue.