I’m having trouble with executing my AWS commands on a gitlab runner. I’m running docker in docker. I then do pip install aws cli. THIS WORKS FOR MY DEV DEPLOY, but not when i try to push to production. So weird. Any Ideas?
A snippet of the log:
Pulling docker image docker:latest ...
Using docker image sha256:bc6c0ffef6650bcfbb0afd5a07b813b5ccf1d00ecddccadb85123c6ee57a7995 for docker:latest with digest docker@sha256:63107bd6ce718f130bb2668704239467b74f034c446f9e9c4ae1ffa5ddf4e3dd ...
Preparing environment
00:03
Running on runner-72989761-project-16409159-concurrent-0 via runner-72989761-srm-1631754494-298283b3...
Getting source from Git repository
00:02
$ eval "$CI_PRE_CLONE_SCRIPT"
Fetching changes with git depth set to 50...
Initialized empty Git repository in /builds/calcutrack/calcutrack-integration/.git/
Created fresh repository.
Checking out 5bf14e16 as 2.8.5...
Skipping Git submodules setup
Downloading artifacts
00:03
Downloading artifacts for maven-build-prod (1594531597)...
Downloading artifacts from coordinator... ok id=1594531597 responseStatus=200 OK token=z--aAj77
Executing "step_script" stage of the job script
00:21
Using docker image sha256:bc6c0ffef6650bcfbb0afd5a07b813b5ccf1d00ecddccadb85123c6ee57a7995 for docker:latest with digest docker@sha256:63107bd6ce718f130bb2668704239467b74f034c446f9e9c4ae1ffa5ddf4e3dd ...
$ apk add --no-cache curl jq python3 py3-pip
fetch https://dl-cdn.alpinelinux.org/alpine/v3.14/main/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.14/community/x86_64/APKINDEX.tar.gz
(1/42) Installing brotli-libs (1.0.9-r5)
(2/42) Installing nghttp2-libs (1.43.0-r0)
(3/42) Installing libcurl (7.79.0-r0)
(4/42) Installing curl (7.79.0-r0)
(5/42) Installing oniguruma (6.9.7.1-r0)
(6/42) Installing jq (1.6-r1)
(7/42) Installing libbz2 (1.0.8-r1)
(8/42) Installing expat (2.4.1-r0)
(9/42) Installing libffi (3.3-r2)
(10/42) Installing gdbm (1.19-r0)
(11/42) Installing xz-libs (5.2.5-r0)
(12/42) Installing libgcc (10.3.1_git20210424-r2)
(13/42) Installing libstdc++ (10.3.1_git20210424-r2)
(14/42) Installing mpdecimal (2.5.1-r1)
(15/42) Installing readline (8.1.0-r0)
(16/42) Installing sqlite-libs (3.35.5-r0)
(17/42) Installing python3 (3.9.5-r1)
(18/42) Installing py3-appdirs (1.4.4-r2)
(19/42) Installing py3-chardet (4.0.0-r2)
(20/42) Installing py3-idna (3.2-r0)
(21/42) Installing py3-urllib3 (1.26.5-r0)
(22/42) Installing py3-certifi (2020.12.5-r1)
(23/42) Installing py3-requests (2.25.1-r4)
(24/42) Installing py3-msgpack (1.0.2-r1)
(25/42) Installing py3-lockfile (0.12.2-r4)
(26/42) Installing py3-cachecontrol (0.12.6-r1)
(27/42) Installing py3-colorama (0.4.4-r1)
(28/42) Installing py3-contextlib2 (0.6.0-r1)
(29/42) Installing py3-distlib (0.3.1-r3)
(30/42) Installing py3-distro (1.5.0-r3)
(31/42) Installing py3-six (1.15.0-r1)
(32/42) Installing py3-webencodings (0.5.1-r4)
(33/42) Installing py3-html5lib (1.1-r1)
(34/42) Installing py3-parsing (2.4.7-r2)
(35/42) Installing py3-packaging (20.9-r1)
(36/42) Installing py3-toml (0.10.2-r2)
(37/42) Installing py3-pep517 (0.10.0-r2)
(38/42) Installing py3-progress (1.5-r2)
(39/42) Installing py3-retrying (1.3.3-r1)
(40/42) Installing py3-ordered-set (4.0.2-r1)
(41/42) Installing py3-setuptools (52.0.0-r3)
(42/42) Installing py3-pip (20.3.4-r1)
Executing busybox-1.33.1-r3.trigger
OK: 84 MiB in 64 packages
$ pip install awscli
Collecting awscli
Downloading awscli-1.20.42-py3-none-any.whl (3.7 MB)
Collecting docutils<0.16,>=0.10
Downloading docutils-0.15.2-py3-none-any.whl (547 kB)
Collecting botocore==1.21.42
Downloading botocore-1.21.42-py3-none-any.whl (7.9 MB)
Collecting rsa<4.8,>=3.1.2
Downloading rsa-4.7.2-py3-none-any.whl (34 kB)
Collecting colorama<0.4.4,>=0.2.5
Downloading colorama-0.4.3-py2.py3-none-any.whl (15 kB)
Collecting s3transfer<0.6.0,>=0.5.0
Downloading s3transfer-0.5.0-py3-none-any.whl (79 kB)
Collecting PyYAML<5.5,>=3.10
Downloading PyYAML-5.4.1.tar.gz (175 kB)
Installing build dependencies: started
Installing build dependencies: finished with status 'done'
Getting requirements to build wheel: started
Getting requirements to build wheel: finished with status 'done'
Preparing wheel metadata: started
Preparing wheel metadata: finished with status 'done'
Collecting jmespath<1.0.0,>=0.7.1
Downloading jmespath-0.10.0-py2.py3-none-any.whl (24 kB)
Collecting python-dateutil<3.0.0,>=2.1
Downloading python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
Requirement already satisfied: urllib3<1.27,>=1.25.4 in /usr/lib/python3.9/site-packages (from botocore==1.21.42->awscli) (1.26.5)
Requirement already satisfied: six>=1.5 in /usr/lib/python3.9/site-packages (from python-dateutil<3.0.0,>=2.1->botocore==1.21.42->awscli) (1.15.0)
Collecting pyasn1>=0.1.3
Downloading pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Building wheels for collected packages: PyYAML
Building wheel for PyYAML (PEP 517): started
Building wheel for PyYAML (PEP 517): finished with status 'done'
Created wheel for PyYAML: filename=PyYAML-5.4.1-cp39-cp39-linux_x86_64.whl size=45654 sha256=52da232e5214c8034a7c1617b426a9be6127ce6c53e508669580e997d5e31170
Stored in directory: /root/.cache/pip/wheels/b7/a5/c4/504d913c2a55bb09c607541578ec5f844d1ff33467abe93ba5
Successfully built PyYAML
Installing collected packages: python-dateutil, jmespath, pyasn1, botocore, s3transfer, rsa, PyYAML, docutils, colorama, awscli
Attempting uninstall: colorama
Found existing installation: colorama 0.4.4
Uninstalling colorama-0.4.4:
Successfully uninstalled colorama-0.4.4
Successfully installed PyYAML-5.4.1 awscli-1.20.42 botocore-1.21.42 colorama-0.4.3 docutils-0.15.2 jmespath-0.10.0 pyasn1-0.4.8 python-dateutil-2.8.2 rsa-4.7.2 s3transfer-0.5.0
$ aws configure set aws_access_key_id $AWS_ACCESS_KEY_ID
Note: AWS CLI version 2, the latest major version of the AWS CLI, is now stable and recommended for general use. For more information, see the AWS CLI version 2 installation instructions at: https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html
usage: aws [options] <command> <subcommand> [<subcommand> ...] [parameters]
To see help text, you can run:
aws help
aws <command> help
aws <command> <subcommand> help
aws: error: the following arguments are required: value
Cleaning up file based variables
00:00
ERROR: Job failed: exit code 2
This is what I have for Prod deploy in my gitlab-ci.yml. Doesn’t Work
deploy-prod:
stage: deploy
image:
name: docker:latest
services:
- docker:19-dind
before_script:
- apk add --no-cache curl jq python3 py3-pip
- pip install awscli
- aws configure set aws_access_key_id $AWS_ACCESS_KEY_ID
- aws configure set aws_secret_access_key $AWS_SECRET_ACCESS_KEY
- aws configure set region $AWS_DEFAULT_REGION
- aws ecr get-login-password --region us-west-2 | docker login $CI_REGISTRY -u AWS
--password-stdin
- aws --version
- docker info
- docker --version
For comparison this is what I have for DEV deploy in my gitlab-ci.yml. It Works
deploy-dev:
stage: deploy
image:
name: docker:latest
services:
- docker:19-dind
before_script:
- apk add --no-cache curl jq python3 py3-pip
- pip install awscli
- aws configure set aws_access_key_id $AWS_ACCESS_KEY_ID
- aws configure set aws_secret_access_key $AWS_SECRET_ACCESS_KEY
- aws configure set region $AWS_DEFAULT_REGION
- aws ecr get-login-password --region us-west-2 | docker login $CI_REGISTRY -u
AWS --password-stdin
- aws --version
- docker info
- docker --version