Describe your question in as much detail as possible:
In my AWS SAM build-and-package step, the docker container fails to pull an image of python3.9. This issue suddenly started happening, and was note occurring in previous builds. Not sure if it’s an issue w/ docker containers, or not?
-
What are you seeing, and how does that differ from what you expect to see?
I’m expecting the build to be successful, like below: -
Consider including screenshots, error messages, and/or other helpful visuals
Below is what I’m currently seeing:
$ python3 --version
Python 3.8.10
$ sam build --template ${SAM_TEMPLATE} --use-container
SAM CLI now collects telemetry to better understand customer needs.
You can OPT OUT and disable telemetry collection by setting the
environment variable SAM_CLI_TELEMETRY=0 in your shell.
Thanks for your help!
Learn More: https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-telemetry.html
Starting Build inside a container
Building codeuri: /builds/qzeng96/vibes/functions/statemachine/all_users runtime: python3.9 metadata: {} architecture: x86_64 functions: ['QueueAllUsersWhoNeedMatchingFunction']
Traceback (most recent call last):
File "/usr/lib/python3.8/site-packages/docker/api/client.py", line 261, in _raise_for_status
response.raise_for_status()
File "/usr/lib/python3.8/site-packages/requests/models.py", line 943, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://docker:2376/v1.35/images/create?tag=latest-x86_64&fromImage=public.ecr.aws%2Fsam%2Fbuild-python3.9%3Alatest-x86_64
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.8/site-packages/samcli/local/docker/manager.py", line 160, in pull_image
result_itr = self.docker_client.api.pull(image_name, tag=tag, stream=True, decode=True)
File "/usr/lib/python3.8/site-packages/docker/api/image.py", line 415, in pull
self._raise_for_status(response)
File "/usr/lib/python3.8/site-packages/docker/api/client.py", line 263, in _raise_for_status
raise create_api_error_from_http_exception(e)
File "/usr/lib/python3.8/site-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
raise cls(e, response=response, explanation=explanation)
docker.errors.NotFound: 404 Client Error: Not Found ("manifest for public.ecr.aws/sam/build-python3.9:latest-x86_64 not found: manifest unknown: Requested image not found")
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/lib/python3.8/site-packages/samcli/local/docker/manager.py", line 85, in create
self.pull_image(image_name)
File "/usr/lib/python3.8/site-packages/samcli/local/docker/manager.py", line 163, in pull_image
raise DockerImagePullFailedException(str(ex)) from ex
samcli.local.docker.manager.DockerImagePullFailedException: 404 Client Error: Not Found ("manifest for public.ecr.aws/sam/build-python3.9:latest-x86_64 not found: manifest unknown: Requested image not found")
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/bin/sam", line 8, in <module>
sys.exit(cli())
File "/usr/lib/python3.8/site-packages/click/core.py", line 829, in __call__
return self.main(*args, **kwargs)
File "/usr/lib/python3.8/site-packages/click/core.py", line 782, in main
rv = self.invoke(ctx)
File "/usr/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/lib/python3.8/site-packages/click/core.py", line 610, in invoke
return callback(*args, **kwargs)
File "/usr/lib/python3.8/site-packages/click/decorators.py", line 73, in new_func
return ctx.invoke(f, obj, *args, **kwargs)
File "/usr/lib/python3.8/site-packages/click/core.py", line 610, in invoke
return callback(*args, **kwargs)
File "/usr/lib/python3.8/site-packages/samcli/lib/telemetry/metric.py", line 153, in wrapped
raise exception # pylint: disable=raising-bad-type
File "/usr/lib/python3.8/site-packages/samcli/lib/telemetry/metric.py", line 122, in wrapped
return_value = func(*args, **kwargs)
File "/usr/lib/python3.8/site-packages/samcli/lib/utils/version_checker.py", line 42, in wrapped
actual_result = func(*args, **kwargs)
File "/usr/lib/python3.8/site-packages/samcli/cli/main.py", line 90, in wrapper
return func(*args, **kwargs)
File "/usr/lib/python3.8/site-packages/samcli/commands/build/command.py", line 210, in cli
do_cli(
File "/usr/lib/python3.8/site-packages/samcli/commands/build/command.py", line 318, in do_cli
artifacts = builder.build()
File "/usr/lib/python3.8/site-packages/samcli/lib/build/app_builder.py", line 172, in build
return build_strategy.build()
File "/usr/lib/python3.8/site-packages/samcli/lib/build/build_strategy.py", line 42, in build
result.update(self._build_functions(self._build_graph))
File "/usr/lib/python3.8/site-packages/samcli/lib/build/build_strategy.py", line 53, in _build_functions
function_build_results.update(self.build_single_function_definition(build_definition))
File "/usr/lib/python3.8/site-packages/samcli/lib/build/build_strategy.py", line 120, in build_single_function_definition
result = self._build_function(
File "/usr/lib/python3.8/site-packages/samcli/lib/build/app_builder.py", line 563, in _build_function
return self._build_function_on_container(
File "/usr/lib/python3.8/site-packages/samcli/lib/build/app_builder.py", line 693, in _build_function_on_container
self._container_manager.run(container)
File "/usr/lib/python3.8/site-packages/samcli/local/docker/manager.py", line 115, in run
self.create(container)
File "/usr/lib/python3.8/site-packages/samcli/local/docker/manager.py", line 88, in create
raise DockerImagePullFailedException(
samcli.local.docker.manager.DockerImagePullFailedException: Could not find public.ecr.aws/sam/build-python3.9:latest-x86_64 image locally and failed to pull it from docker.
-
What version are you on? Are you using self-managed or GitLab.com?
-
GitLab (Hint:
/help
): Yes, on Gitlab.com -
Runner (Hint:
/admin/runners
): Shared runners for this project is enabled
-
GitLab (Hint:
-
Add the CI configuration from
.gitlab-ci.yml
and other configuration if relevant (e.g. docker-compose.yml)
https://gitlab.com/qzeng96/vibes/-/blob/main/.gitlab-ci.yml
variables:
SAM_TEMPLATE: template.yaml
PIPELINE_USER_ACCESS_KEY_ID: ${AWS_ACCESS_KEY_ID}
PIPELINE_USER_SECRET_ACCESS_KEY: ${AWS_SECRET_ACCESS_KEY}
TESTING_STACK_NAME: vibes-development-stack
TESTING_REGION: us-east-1
TESTING_PIPELINE_EXECUTION_ROLE: arn:aws:iam::100620893444:role/aws-sam-cli-managed-vibes-de-PipelineExecutionRole-1M9EK0TWBI2TJ
TESTING_CLOUDFORMATION_EXECUTION_ROLE: arn:aws:iam::100620893444:role/aws-sam-cli-managed-vibes-CloudFormationExecutionR-1438AXEW3LK4K
TESTING_ARTIFACTS_BUCKET: aws-sam-cli-managed-vibes-develop-artifactsbucket-15iecr0p1ni0q
# If there are functions with "Image" PackageType in your template,
# uncomment the line below and add "--image-repository ${TESTING_IMAGE_REPOSITORY}" to
# testing "sam package" and "sam deploy" commands.'
# TESTING_IMAGE_REPOSITORY = '0123456789.dkr.ecr.region.amazonaws.com/repository-name'
PROD_STACK_NAME: vibes-production-stack
PROD_REGION: us-east-1
PROD_PIPELINE_EXECUTION_ROLE: arn:aws:iam::456534673314:role/aws-sam-cli-managed-vibes-pr-PipelineExecutionRole-Y7PYDNJRZDR4
PROD_CLOUDFORMATION_EXECUTION_ROLE: arn:aws:iam::456534673314:role/aws-sam-cli-managed-vibes-CloudFormationExecutionR-1D9SX1NJOJCIR
PROD_ARTIFACTS_BUCKET: aws-sam-cli-managed-vibes-product-artifactsbucket-os58krj9lev9
# If there are functions with "Image" PackageType in your template,
# uncomment the line below and add "--image-repository ${PROD_IMAGE_REPOSITORY}" to
# prod "sam package" and "sam deploy" commands.'
# PROD_IMAGE_REPOSITORY = '0123456789.dkr.ecr.region.amazonaws.com/repository-name'
# By default, when using docker:dind, Docker uses the vfs storage
# driver which copies the file system on every run.
# This is a disk-intensive operation which can be avoided if a different driver is used.
# For example overlay2
DOCKER_DRIVER: overlay2
# Create the certificates inside this directory for both the server
# and client. The certificates used by the client will be created in
# /certs/client so we only need to share this directory with the
# volume mount in `config.toml`.
DOCKER_TLS_CERTDIR: "/certs"
# Should always specify a specific version of the image. If using a tag like docker:stable,
# there will be no control over which version is used. Unpredictable behavior can result.
image: docker:20.10.6
services:
- docker:20.10.6-dind
before_script:
- apk add --update python3 py-pip python3-dev build-base
- pip install awscli aws-sam-cli
- python3 --version
stages:
- unit-test
- build
- testing
- prod
# uncomment and modify the following step for running the unit-tests
#
#unit-test:
# stage: unit-test
# only:
# - main
# - /^feature-.*$/
# script: |
# This stage is triggered only for feature branches (feature*),
# which will build the stack and deploy to a stack named with branch name.
build-and-deploy-feature:
stage: build
only:
- /^feature-.*$/
script:
- . assume-role.sh ${TESTING_PIPELINE_EXECUTION_ROLE} feature-deployment
- sam build --template ${SAM_TEMPLATE} --use-container
- sam deploy --stack-name $(echo ${CI_COMMIT_REF_NAME} | tr -cd '[a-zA-Z0-9-]')
--capabilities CAPABILITY_IAM
--region ${TESTING_REGION}
--s3-bucket ${TESTING_ARTIFACTS_BUCKET}
--no-fail-on-empty-changeset
--role-arn ${TESTING_CLOUDFORMATION_EXECUTION_ROLE}
# This stage is triggered for main branch you set in the question,
# which will build the stack, package the application, upload the
# applications artifacts to Amazon S3 and output the SAM template file.
build-and-package:
stage: build
only:
- main
script:
- sam build --template ${SAM_TEMPLATE} --use-container
- . assume-role.sh ${TESTING_PIPELINE_EXECUTION_ROLE} testing-stage-packaging
- sam package --s3-bucket ${TESTING_ARTIFACTS_BUCKET}
--region ${TESTING_REGION}
--output-template-file packaged-testing.yaml
- . assume-role.sh ${PROD_PIPELINE_EXECUTION_ROLE} prod-stage-packaging
- sam package --s3-bucket ${PROD_ARTIFACTS_BUCKET}
--region ${PROD_REGION}
--output-template-file packaged-prod.yaml
artifacts:
paths:
- packaged-testing.yaml
- packaged-prod.yaml
# This stage is triggered for main branch you set in the question,
# which will deploy the testing stage SAM application using
# the templated file generated.
deploy-testing:
stage: testing
only:
- main
script:
- . assume-role.sh ${TESTING_PIPELINE_EXECUTION_ROLE} testing-deployment
- sam deploy --stack-name ${TESTING_STACK_NAME}
--template packaged-testing.yaml
--capabilities CAPABILITY_IAM
--region ${TESTING_REGION}
--s3-bucket ${TESTING_ARTIFACTS_BUCKET}
--no-fail-on-empty-changeset
--role-arn ${TESTING_CLOUDFORMATION_EXECUTION_ROLE}
# Uncomment and modify the following stage for integration tests
#
#integration-test:
# stage: testing
# only:
# - main
# script: |
# #trigger the integration tests here
# This stage is triggered for main branch you set in the question,
# which will deploy the prod stage SAM application using
# the templated file generated.
deploy-prod:
stage: prod
# uncomment this to have a manual approval step before deployment to production
# when: manual
only:
- main
script:
- . assume-role.sh ${PROD_PIPELINE_EXECUTION_ROLE} prod-deployment
- sam deploy --stack-name ${PROD_STACK_NAME}
--template packaged-prod.yaml
--capabilities CAPABILITY_IAM
--region ${PROD_REGION}
--s3-bucket ${PROD_ARTIFACTS_BUCKET}
--no-fail-on-empty-changeset
--role-arn ${PROD_CLOUDFORMATION_EXECUTION_ROLE}
-
What troubleshooting steps have you already taken? Can you link to any docs or other resources so we know where you have been?
- Attempted changing
image
to python:3.9, rather than the current value.
- Attempted changing
Thanks for taking the time to be thorough in your request, it really helps!