Gitlab CI (Docker Runner) - Terraform Permission denied

Hi guys,

i want to run terraform code in a gitlab ci docker runner. We use gitlab on prem.
I followed this template:
https://docs.gitlab.com/ee/user/infrastructure/index.html

After terraform init when i want to execute the code (terraform plan or apply) under the build folder i get the following:

[43](https://gitlabPATH/-/jobs/12697#L43)Error: Failed to instantiate provider "gitlab" to obtain schema: fork/exec /builds/<projectPATH>/.terraform/plugins/linux_amd64/terraform-provider-gitlab_v2.11.0_x4: permission denied

[44](https://gitlabPATH/-/jobs/12697#L44)Error: Failed to instantiate provider "local" to obtain schema: fork/exec /builds/<projectPATH>/.terraform/plugins/linux_amd64/terraform-provider-local_v1.4.0_x4: permission denied

[45](https://gitlabPATH/-/jobs/12697#L45)Error: Failed to instantiate provider "null" to obtain schema: fork/exec /builds/<projectPATH>/.terraform/plugins/linux_amd64/terraform-provider-null_v2.1.2_x4: permission denied

This happens because the project files are mounted into the container without execution rights under the builds path. To work around this i copy the complete folder out of /builds but that is not nice.

Is there a way to enable the execution in the /builds folder (but that seems not to be the right way)? In all your tutorials you have never the problem so do i do something wrong or do i have missed something?

I would be really happy if someone could help me. I cant believe that im the only one with that problem.

Best regards

Stefan

Hey, Stefan.

I’m not sure I have understand what are you trying to do with /builds directory.

I’ve been using terraform pipelines for a while and didn’t saw this issue yet.

Could you provide more details about the steps are you trying to execute?

Thanks!

Hi ayr-ton,

sure
You can get test code here:

The part that is of intereset is everything after the init so e.g. validate.
When i run it on gitlab.com via docker maschine executer everything works and looks like this:

Running with gitlab-runner 13.3.0-rc1 (669fc507)
  on docker-auto-scale 0277ea0f
Preparing the "docker+machine" executor
00:14
Preparing environment
00:01
Running on runner-0277ea0f-project-20748756-concurrent-0 via runner-0277ea0f-srm-1598428699-b801c9e8...
Getting source from Git repository
00:01
$ eval "$CI_PRE_CLONE_SCRIPT"
Fetching changes with git depth set to 50...
Initialized empty Git repository in /builds/seamex/test-terraform/.git/
Created fresh repository.
Checking out 660ff1f4 as master...
Skipping Git submodules setup
Restoring cache
00:02
Checking cache for example...
Downloading cache.zip from https://storage.googleapis.com/gitlab-com-runners-cache/project/20748756/example 
Successfully extracted cache
Executing "step_script" stage of the job script
00:01
$ cd ${TF_ROOT}
$ gitlab-terraform validate
+ plan_cache=plan.cache
+ plan_json=plan.json
+ JQ_PLAN='
  (
    [.resource_changes[]?.change.actions?] | flatten
  ) | {
    "create":(map(select(.=="create")) | length),
    "update":(map(select(.=="update")) | length),
    "delete":(map(select(.=="delete")) | length)
  }
'
+ '[' -z  ]
+ TF_USERNAME=seamex
+ '[' -z  ]
+ TF_USERNAME=gitlab-ci-token
+ TF_PASSWORD=[MASKED]
+ terraform validate
Success! The configuration is valid.
Saving cache
00:02
Creating cache example...
/builds/seamex/test-terraform/.terraform: found 5 matching files and directories 
Uploading cache.zip to https://storage.googleapis.com/gitlab-com-runners-cache/project/20748756/example 
Created cache
Job succeeded

When i run it via docker runner it dont work and looks like this:
Running with gitlab-runner 13.2.1 (efa30e33)
on host Jf6hXwRK
Preparing the “docker” executor
00:03
Preparing environment
00:00
Running on runner-jf6hxwrk-project-311-concurrent-0 via host…
Getting source from Git repository
00:01
Fetching changes with git depth set to 50…
Reinitialized existing Git repository in /builds/mertenss/test-terraform/.git/
Checking out 660ff1f4 as master…
Removing .terraform/
Skipping Git submodules setup
Restoring cache
00:01
Checking cache for example…
No URL provided, cache will not be downloaded from shared cache server. Instead a local version of cache will be extracted.
Successfully extracted cache
Executing “step_script” stage of the job script
00:01
cd {TF_ROOT}
$ gitlab-terraform validate
+ plan_cache=plan.cache
+ plan_json=plan.json
+ JQ_PLAN=’
(
[.resource_changes?.change.actions?] | flatten
) | {
“create”:(map(select(.==“create”)) | length),
“update”:(map(select(.==“update”)) | length),
“delete”:(map(select(.==“delete”)) | length)
}
’
+ ‘[’ -z ]
+ TF_USERNAME=mertenss
+ ‘[’ -z ]
+ TF_USERNAME=gitlab-ci-token
+ TF_PASSWORD=[MASKED]
+ terraform validate
Error: Failed to instantiate provider “local” to obtain schema: fork/exec /builds/mertenss/test-terraform/.terraform/plugins/linux_amd64/terraform-provider-local_v1.4.0_x4: permission denied
ERROR: Job failed: exit code 1

The permission denied is a failure because the git repo is mounted in the container without execution rights. The question is how to enable that execution right or what have i done wrong?

This is my runner config:

concurrent = 8
check_interval = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "host"
  url = "https://urltoonpremgitlab/"
  token = "token"
  executor = "docker"
  [runners.custom_build_dir]
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]
  [runners.docker]
    tls_verify = false
    image = "alpine:latest"
    privileged = true
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/var/run/docker.sock:/var/run/docker.sock", "/cache" ]
    shm_size = 0

I found my problem. We install our docker on a separate lv and that was mounted via relatime,noexec

This is one flow that I have used sometimes on the past:

Today I use a similar one with terragrunt for applying different modules and accounts, but the main pipeline idea is the same. I could publish the code as well later for reference.