Gitlab CICD & AWS CDK Deploy - Unable to determine the default AWS account: IncompleteSignature

Situtation:
When I cdk deploy locally, I can deploy the stack to the correct account. However, when via gitlab-ci.yml, I get the error: Error: Need to perform AWS calls for account XXXXXX, but no credentials have been configured

  • What are you seeing, and how does that differ from what you expect to see?
    I would expect that when I set the env vars in the Gitlab UI for the access key and secret key, cdk deploy would find the creds in ~/.aws/. Yet this don’t seem to be the case.

I’ve listed at the bottom the two reference forums I used but still haven’t figured out why cdk deploy won’t read the credentials correctly.

Screenshots, error messages, other helpful visuals*

Output from Gitlab-CI job run showing keys are set for default:

$ aws configure set aws_access_key_id "$AWS_ACCESS_KEY_ID"
$ aws configure set aws_secret_access_key "$AWS_SECRET_ACCESS_KEY"
$ aws configure list
      Name                    Value             Type    Location
      ----                    -----             ----    --------
   profile                <not set>             None    None
access_key         **************** shared-credentials-file    
secret_key         **************** shared-credentials-file 
#[omitted.......]
Determining if we're on an EC2 instance.
Does not look like an EC2 instance.
Unable to determine AWS region from environment or AWS configuration (profile: "default"), defaulting to 'us-east-1'
Toolkit stack: CDKToolkit
Setting "CDK_DEFAULT_REGION" environment variable to us-east-1
Resolving default credentials
Looking up default account ID from STS
Unable to determine the default AWS account: IncompleteSignature: '/20210920/us-east-1/sts/aws4_request' not a valid key=value pair (missing equal-sign) in Authorization header: 'AWS4-HMAC-SHA256 Credential= /20210920/us-east-1/sts/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=e30ef933fae1de698850a1a75b3e258b5263c480497d2587727861a7e655e59b'.
    at Request.extractError (/usr/local/lib/node_modules/aws-cdk/node_modules/aws-sdk/lib/protocol/query.js:50:29)
    at Request.callListeners (/usr/local/lib/node_modules/aws-cdk/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
    at Request.emit (/usr/local/lib/node_modules/aws-cdk/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
    at Request.emit (/usr/local/lib/node_modules/aws-cdk/node_modules/aws-sdk/lib/request.js:688:14)
    at Request.transition (/usr/local/lib/node_modules/aws-cdk/node_modules/aws-sdk/lib/request.js:22:10)
    at AcceptorStateMachine.runTo (/usr/local/lib/node_modules/aws-cdk/node_modules/aws-sdk/lib/state_machine.js:14:12)
    at /usr/local/lib/node_modules/aws-cdk/node_modules/aws-sdk/lib/state_machine.js:26:10
    at Request.<anonymous> (/usr/local/lib/node_modules/aws-cdk/node_modules/aws-sdk/lib/request.js:38:9)
    at Request.<anonymous> (/usr/local/lib/node_modules/aws-cdk/node_modules/aws-sdk/lib/request.js:690:12)
    at Request.callListeners (/usr/local/lib/node_modules/aws-cdk/node_modules/aws-sdk/lib/sequential_executor.js:116:18) {
  code: 'IncompleteSignature',
  time: 2021-09-20T18:17:12.943Z,
  requestId: 'cdd5e0fa-20ac-4a97-ac08-2870246eecbd',
  statusCode: 400,
  retryable: false,
  retryDelay: 117.36976174983293
}


Gitlab-ci.yml

The account number is hardcode for the time being in app.py, fyi.
Using GitLab Community Edition 13.6.3

ev-build-deploy-job:
  stage: dev
  rules: 
    - if: '$CI_COMMIT_BRANCH == "dev"'
  script: 
    - echo "Installing packages..."
    - apt-get -qq update && apt-get -y install nodejs npm
    - node -v
    - npm i -g aws-cdk
    - pip3 install awscli
    - pip3 install -r requirements.txt
    - npm --version 
    - mkdir ~/.aws/ 
    - touch ~/.aws/credentials 
    - touch ~/.aws/config 
    - aws configure set aws_access_key_id "$AWS_ACCESS_KEY_ID"
    - aws configure set aws_secret_access_key "$AWS_SECRET_ACCESS_KEY"
    - aws configure list
    - cdk deploy --require-approval never --verbose
  environment: 
    name: dev/$CI_COMMIT_REF_NAME

Troubleshooting steps Taken?

The above cmds match what folks seem to be using in the forums below.

  1. amazon web services - Gitlab CI / CD pipeline authentication error to AWS S3 public bucket - upload failed:a non-empty Access Key (AKID) must be provided in the credential - Stack Overflow
  2. amazon web services - What is the best way to do CI/CD with AWS CDK (python) using GitLab CI? - Stack Overflow

Thanks to a prompt from Brian Caffey, I didn’t give/forgot to give the dev branch access to the creds in Gitlab.

Once I set the dev branch to ‘protected’, everything works as expected.