Authenticate to AWS using CI_JOB_TOKEN

GitLab CI/CD
1

Hello,

IAM is able to delegate authentication to an external identity provider through OIDC (Creating OpenID Connect (OIDC) identity providers - AWS Identity and Access Management).

I tried to define gitlab as an external Identity provider in order to be able to authenticate to IAM using CI_JOB_TOKEN.

I created an IAM role with OIDC, then set AWS_ROLE_ARN and dump CI_JOB_TOKEN to a file and point AWS_WEB_IDENTITY_TOKEN_FILE to it.

When I run aws sts assume-role-with-web-identity --role-arn $AWS_ROLE_ARN --role-session-name test --web-identity-token file://$AWS_WEB_IDENTITY_TOKEN_FILE --duration-seconds 1000, I get this error: An error occurred (InvalidIdentityToken) when calling the AssumeRoleWithWebIdentity operation: Missing a required claim: aud.

Indeed, CI_JOB_TOKEN does not contain any aud claim.

My goal is to achieve this: Authenticating and reading secrets with HashiCorp Vault | GitLab but without having to deploy HashCorp Vault.

Is there something I misunderstand in OIDC (which is totally possible) and what I’m trying to do is completely dumb?

It would be nice to be able to authenticate Gitlab CI jobs towards IAM without having to deploy Hashicorp Vault.

2

When trying to understand how IAM Role for ServiceAccount works, I decoded a ServiceAccount token, here is what it looks like:

{
  "aud": [
    "https://kubernetes.default.svc.cluster.local"
  ],
  "exp": 1664781927,
  "iat": 1633245927,
  "iss": "https://kubernetes.default.svc.cluster.local",
  "kubernetes.io": {
    "namespace": "kube-system",
    "pod": {
      "name": "kindnet-xrlgj",
      "uid": "314bd99e-c985-48f6-94c0-e19273dc75f8"
    },
    "serviceaccount": {
      "name": "kindnet",
      "uid": "dc206c6d-334f-4994-a1f0-6ab50c20dc1b"
    },
    "warnafter": 1633249534
  },
  "nbf": 1633245927,
  "sub": "system:serviceaccount:kube-system:kindnet"
}

While here is what a GitLab job token looks like:

{
  "namespace_id": "123456",
  "namespace_path": "<username>",
  "project_id": "67890",
  "project_path": "<username>/<project>",
  "user_id": "87398753",
  "user_login": "<username>",
  "user_email": "<email>",
  "pipeline_id": "98080980",
  "pipeline_source": "merge_request_event",
  "job_id": "1387684868",
  "ref": "<branch>",
  "ref_type": "branch",
  "ref_protected": "false",
  "jti": "7f945784-8081-4909-bd97-eb0f7df6dddf",
  "iss": "gitlab.com",
  "iat": 1633184303,
  "nbf": 1633184298,
  "exp": 1633187903,
  "sub": "job_1387684868"
}

I guess IAM uses aud to filter out unauthorized tokens and sub to identify the ServiceAccount. So even if CI_JOB_TOKEN would have a properly set aud claim, I would have an issue identifying the job because the sub contains the job_id which is a sequence.

I guess it could work if the sub would be something like: <project_path> or maybe something more complexe like <project_path>:<pipeline_source> so that we could give different permissions to merge requests and commits.

Or maybe the solution would be that IAM is more flexible and allows to identify on something else than sub?

3

I patched my GitLab instance with this patch: Feat(ci): add aud claim to CI_JOB_TOKEN (!71657) · Merge requests · GitLab.org / GitLab · GitLab,

Now the error message is: `An error occurred (InvalidIdentityToken) when calling the AssumeRoleWithWebIdentity operation: Couldn’t retrieve verification key from your identity provider, please reference AssumeRoleWithWebIdentity documentation for requirements``

I can’t figure out what happens now…

Any idea welcome

4

Here’s a description of GitHub’s recently added support for this flow: AWS federation comes to GitHub Actions | Aidan Steele’s blog (usually about AWS)

5

Here is an issue I created trying to explain my journey: Allow CI_JOB_TOKEN to AssumeRoleWithWebIdentity in AWS IAM (#342243) · Issues · GitLab.org / GitLab · GitLab

6

OK, so what Github do is setting repo_name in sub. I guess this would fix my last issue.

7

@mcanevet I’m curious if you got this to work. At my company the runner itself has an instance profile which allows it to assume a role in the target account where the actions are to be performed. I am wondering if you got this to work and what your experience with this pattern has been.

edit for clarity - Our runners are hosted on EC2 instances. It was not clear to me where your runners are (or if you are using shared runners)

8

Long story short: it is not possible for now because of 4 reasons: Allow CI_JOB_TOKEN to AssumeRoleWithWebIdentity in AWS IAM (#342243) · Issues · GitLab.org / GitLab · GitLab

9

Hi i am using that one to authorize my gitlab jobs to access aws: LittleStartup / SmartContextProvider · GitLab