Unable to get Premium Vault integration working [solved - unrelated to GitLab]

I have been unable to get vault integration working in our CI pipeline. We are premium customers using gitlab.com. I have followed the instructions here: Using external secrets in CI | GitLab. No matter what I try I end up with a 403:

Running with gitlab-runner 15.1.0~beta.20.g62206bb2 (62206bb2)
  on green-2.shared.runners-manager.gitlab.com/default ns46NMmJ
Resolving secrets
00:00
Resolving secret "DB_PASSWORD"...
Using "vault" secret resolver...
ERROR: Job failed (system failure): resolving secrets: reading secret: reading from Vault: api error: status code 403: 1 error occurred:
	* permission denied

Vault address and roles are all set in CI variables, these are tested and work fine if we do this manually in the script section.

I have tried it both with and without a variable in the vault path.

  secrets:
    DB_PASSWORD:
      vault: group_name/project_name/$CI_ENVIRONMENT_TIER/db/password@secret
      file: false

  secrets:
    DB_PASSWORD:
      vault: group_name/project_name/development/db/password@secret
      file: false

Claims, roles, policies, and all of this are correct and work when done manually via script.

{
   "role_type": "jwt",
    "policies": ["ci", "development"],
    "token_explicit_max_ttl": 60,
    "user_claim": "user_email",
    "bound_claims": {
        "project_id": "123456"
    },
    "claim_mappings": {
        "project_path": "project_path"
    }
}

path "secret/data/{{identity.entity.aliases.jwt.metadata.project_path}}/development/*" {
  capabilities = [ "read" ]
}

Without the ability to view the JWT and API call that GitLab is using I cannot troubleshoot this any further. Any guidance would be appreciated.

As you are a premium customer, you would probably get faster support by opening a ticket: https://support.gitlab.com/hc/en-us/requests/new

The Gitlab Team will respond to paying customers far quicker this way, than community support on the forums.

The 403 failure is coming from your hashicorp vault service, Have you checked the logs on that side to see why its rejecting the authentication?

After several days of troubleshooting this and getting nowhere we made this post hoping someone might know how to debug the darn thing better without actually seeing the JWT…turns out making a forum post is a perfect rubber duck.

Issue was solved and unrelated to GItLab. We had an oversight in our Vault policy. Thanks for at least trying to help.

1 Like