Vault external secrets "permission denied" error with CI JWT

GitLab CI/CD
1

I am trying to set up external secrets with Vault and I followed this documentation, but the CI job is throwing a “403 permission denied” error from Vault with no extra details, so I don’t know exactly what request it’s failing on.

My Vault structure looks like this:

JWT Backend - /auth/jwt/gitlab

path = jwt/gitlab
jwks_url = https://gitlab.com/-/jwks
bound_issuer = gitlab.com

Policy - my-policy

path "the/secret/path/*" {
  capabilities = ["read"]
}

path "the/secret/path" {
  capabilities = ["read"]
}

Role - /auth/jwt/gitlab/role/my-app


name = "my-app"
role_type = "jwt"
user_claim = "user_email"
token_type = "default"
token_policies = [my-policy]
bound_claims = {
  project_id = "xxxxxx"
}

Entity - my-app

no config

Alias - my-app-ci

mount_accessor = <JWT Backend accessor ID>
canonical_id = <ID of my-app Entity>

Then in the CI job I have:

variables:
  VAULT_SERVER_URL: https://my-vault.com
  VAULT_AUTH_ROLE: my-app
  VAULT_AUTH_PATH: jwt/gitlab
secrets:
  MY_SECRET:
    vault: the/secret/path/KEY@kv

I don’t know how else to figure out what’s wrong since GitLab gives no indication of what it was trying to do when it got the error.

I am on GitLab SaaS Premium.

2

I have the same issue.

I noticed that when you use this policy:

path "*" {
  capabilities = ["read"]
}

the job succeeds. This is obviously no valid workaround but maybe it helps tracking down the issue.

3

Interesting. I feel like this could be easily solved if GitLab just logged the API calls it was trying to make because clearly it’s trying to access more than the specified secret path.

4

I found the solution! My policy had to grant permission to kv/data/the/path/here for some reason. This is very strange because there were other authentication backends using the same policies and they have always worked fine.

This really needs to be documented because I just figured this out after random guessing.