Docker - Could not find python3 image locally and failed to pull it from docker

GitLab CI/CD
, , ,
1

Describe your question in as much detail as possible:
In my AWS SAM build-and-package step, the docker container fails to pull an image of python3.9. This issue suddenly started happening, and was note occurring in previous builds. Not sure if it’s an issue w/ docker containers, or not?

  • What are you seeing, and how does that differ from what you expect to see?
    I’m expecting the build to be successful, like below:

  • Consider including screenshots, error messages, and/or other helpful visuals
    Below is what I’m currently seeing:

$ python3 --version
Python 3.8.10
$ sam build --template ${SAM_TEMPLATE} --use-container
	SAM CLI now collects telemetry to better understand customer needs.
	You can OPT OUT and disable telemetry collection by setting the
	environment variable SAM_CLI_TELEMETRY=0 in your shell.
	Thanks for your help!
	Learn More: https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-telemetry.html
Starting Build inside a container
Building codeuri: /builds/qzeng96/vibes/functions/statemachine/all_users runtime: python3.9 metadata: {} architecture: x86_64 functions: ['QueueAllUsersWhoNeedMatchingFunction']
Traceback (most recent call last):
  File "/usr/lib/python3.8/site-packages/docker/api/client.py", line 261, in _raise_for_status
    response.raise_for_status()
  File "/usr/lib/python3.8/site-packages/requests/models.py", line 943, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://docker:2376/v1.35/images/create?tag=latest-x86_64&fromImage=public.ecr.aws%2Fsam%2Fbuild-python3.9%3Alatest-x86_64
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/lib/python3.8/site-packages/samcli/local/docker/manager.py", line 160, in pull_image
    result_itr = self.docker_client.api.pull(image_name, tag=tag, stream=True, decode=True)
  File "/usr/lib/python3.8/site-packages/docker/api/image.py", line 415, in pull
    self._raise_for_status(response)
  File "/usr/lib/python3.8/site-packages/docker/api/client.py", line 263, in _raise_for_status
    raise create_api_error_from_http_exception(e)
  File "/usr/lib/python3.8/site-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
    raise cls(e, response=response, explanation=explanation)
docker.errors.NotFound: 404 Client Error: Not Found ("manifest for public.ecr.aws/sam/build-python3.9:latest-x86_64 not found: manifest unknown: Requested image not found")
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "/usr/lib/python3.8/site-packages/samcli/local/docker/manager.py", line 85, in create
    self.pull_image(image_name)
  File "/usr/lib/python3.8/site-packages/samcli/local/docker/manager.py", line 163, in pull_image
    raise DockerImagePullFailedException(str(ex)) from ex
samcli.local.docker.manager.DockerImagePullFailedException: 404 Client Error: Not Found ("manifest for public.ecr.aws/sam/build-python3.9:latest-x86_64 not found: manifest unknown: Requested image not found")
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "/usr/bin/sam", line 8, in <module>
    sys.exit(cli())
  File "/usr/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/usr/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/lib/python3.8/site-packages/click/decorators.py", line 73, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
  File "/usr/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/lib/python3.8/site-packages/samcli/lib/telemetry/metric.py", line 153, in wrapped
    raise exception  # pylint: disable=raising-bad-type
  File "/usr/lib/python3.8/site-packages/samcli/lib/telemetry/metric.py", line 122, in wrapped
    return_value = func(*args, **kwargs)
  File "/usr/lib/python3.8/site-packages/samcli/lib/utils/version_checker.py", line 42, in wrapped
    actual_result = func(*args, **kwargs)
  File "/usr/lib/python3.8/site-packages/samcli/cli/main.py", line 90, in wrapper
    return func(*args, **kwargs)
  File "/usr/lib/python3.8/site-packages/samcli/commands/build/command.py", line 210, in cli
    do_cli(
  File "/usr/lib/python3.8/site-packages/samcli/commands/build/command.py", line 318, in do_cli
    artifacts = builder.build()
  File "/usr/lib/python3.8/site-packages/samcli/lib/build/app_builder.py", line 172, in build
    return build_strategy.build()
  File "/usr/lib/python3.8/site-packages/samcli/lib/build/build_strategy.py", line 42, in build
    result.update(self._build_functions(self._build_graph))
  File "/usr/lib/python3.8/site-packages/samcli/lib/build/build_strategy.py", line 53, in _build_functions
    function_build_results.update(self.build_single_function_definition(build_definition))
  File "/usr/lib/python3.8/site-packages/samcli/lib/build/build_strategy.py", line 120, in build_single_function_definition
    result = self._build_function(
  File "/usr/lib/python3.8/site-packages/samcli/lib/build/app_builder.py", line 563, in _build_function
    return self._build_function_on_container(
  File "/usr/lib/python3.8/site-packages/samcli/lib/build/app_builder.py", line 693, in _build_function_on_container
    self._container_manager.run(container)
  File "/usr/lib/python3.8/site-packages/samcli/local/docker/manager.py", line 115, in run
    self.create(container)
  File "/usr/lib/python3.8/site-packages/samcli/local/docker/manager.py", line 88, in create
    raise DockerImagePullFailedException(
samcli.local.docker.manager.DockerImagePullFailedException: Could not find public.ecr.aws/sam/build-python3.9:latest-x86_64 image locally and failed to pull it from docker.

variables:
  SAM_TEMPLATE: template.yaml
  PIPELINE_USER_ACCESS_KEY_ID: ${AWS_ACCESS_KEY_ID}
  PIPELINE_USER_SECRET_ACCESS_KEY: ${AWS_SECRET_ACCESS_KEY}
  TESTING_STACK_NAME: vibes-development-stack
  TESTING_REGION: us-east-1
  TESTING_PIPELINE_EXECUTION_ROLE: arn:aws:iam::100620893444:role/aws-sam-cli-managed-vibes-de-PipelineExecutionRole-1M9EK0TWBI2TJ
  TESTING_CLOUDFORMATION_EXECUTION_ROLE: arn:aws:iam::100620893444:role/aws-sam-cli-managed-vibes-CloudFormationExecutionR-1438AXEW3LK4K
  TESTING_ARTIFACTS_BUCKET: aws-sam-cli-managed-vibes-develop-artifactsbucket-15iecr0p1ni0q
  # If there are functions with "Image" PackageType in your template,
  # uncomment the line below and add "--image-repository ${TESTING_IMAGE_REPOSITORY}" to
  # testing "sam package" and "sam deploy" commands.'
  # TESTING_IMAGE_REPOSITORY = '0123456789.dkr.ecr.region.amazonaws.com/repository-name'
  PROD_STACK_NAME: vibes-production-stack
  PROD_REGION: us-east-1
  PROD_PIPELINE_EXECUTION_ROLE: arn:aws:iam::456534673314:role/aws-sam-cli-managed-vibes-pr-PipelineExecutionRole-Y7PYDNJRZDR4
  PROD_CLOUDFORMATION_EXECUTION_ROLE: arn:aws:iam::456534673314:role/aws-sam-cli-managed-vibes-CloudFormationExecutionR-1D9SX1NJOJCIR
  PROD_ARTIFACTS_BUCKET: aws-sam-cli-managed-vibes-product-artifactsbucket-os58krj9lev9
  # If there are functions with "Image" PackageType in your template,
  # uncomment the line below and add "--image-repository ${PROD_IMAGE_REPOSITORY}" to
  # prod "sam package" and "sam deploy" commands.'
  # PROD_IMAGE_REPOSITORY = '0123456789.dkr.ecr.region.amazonaws.com/repository-name'
  # By default, when using docker:dind, Docker uses the vfs storage
  # driver which copies the file system on every run.
  # This is a disk-intensive operation which can be avoided if a different driver is used.
  # For example overlay2
  DOCKER_DRIVER: overlay2
  # Create the certificates inside this directory for both the server
  # and client. The certificates used by the client will be created in
  # /certs/client so we only need to share this directory with the
  # volume mount in `config.toml`.
  DOCKER_TLS_CERTDIR: "/certs"

# Should always specify a specific version of the image. If using a tag like docker:stable,
# there will be no control over which version is used. Unpredictable behavior can result.
image: docker:20.10.6

services:
    - docker:20.10.6-dind

before_script:
  - apk add --update python3 py-pip python3-dev build-base
  - pip install awscli aws-sam-cli
  - python3 --version

stages:
  - unit-test
  - build
  - testing
  - prod

# uncomment and modify the following step for running the unit-tests
#
#unit-test:
#  stage: unit-test
#  only:
#    - main
#    - /^feature-.*$/
#  script: |

# This stage is triggered only for feature branches (feature*),
# which will build the stack and deploy to a stack named with branch name.
build-and-deploy-feature:
  stage: build
  only:
    - /^feature-.*$/
  script:
    - . assume-role.sh ${TESTING_PIPELINE_EXECUTION_ROLE} feature-deployment
    - sam build --template ${SAM_TEMPLATE} --use-container
    - sam deploy --stack-name $(echo ${CI_COMMIT_REF_NAME} | tr -cd '[a-zA-Z0-9-]')
                 --capabilities CAPABILITY_IAM
                 --region ${TESTING_REGION}
                 --s3-bucket ${TESTING_ARTIFACTS_BUCKET}
                 --no-fail-on-empty-changeset
                 --role-arn ${TESTING_CLOUDFORMATION_EXECUTION_ROLE}

# This stage is triggered for main branch you set in the question,
# which will build the stack, package the application, upload the
# applications artifacts to Amazon S3 and output the SAM template file.
build-and-package:
  stage: build
  only:
    - main
  script:
    - sam build --template ${SAM_TEMPLATE} --use-container

    - . assume-role.sh ${TESTING_PIPELINE_EXECUTION_ROLE} testing-stage-packaging

    - sam package --s3-bucket ${TESTING_ARTIFACTS_BUCKET}
                   --region ${TESTING_REGION}
                   --output-template-file packaged-testing.yaml

    - . assume-role.sh ${PROD_PIPELINE_EXECUTION_ROLE} prod-stage-packaging

    - sam package --s3-bucket ${PROD_ARTIFACTS_BUCKET}
                  --region ${PROD_REGION}
                  --output-template-file packaged-prod.yaml

  artifacts:
    paths:
      - packaged-testing.yaml
      - packaged-prod.yaml

# This stage is triggered for main branch you set in the question,
# which will deploy the testing stage SAM application using
# the templated file generated.
deploy-testing:
  stage: testing
  only:
    - main
  script:
    - . assume-role.sh ${TESTING_PIPELINE_EXECUTION_ROLE} testing-deployment
    - sam deploy --stack-name ${TESTING_STACK_NAME}
                 --template packaged-testing.yaml
                 --capabilities CAPABILITY_IAM
                 --region ${TESTING_REGION}
                 --s3-bucket ${TESTING_ARTIFACTS_BUCKET}
                 --no-fail-on-empty-changeset
                 --role-arn ${TESTING_CLOUDFORMATION_EXECUTION_ROLE}

# Uncomment and modify the following stage for integration tests
#
#integration-test:
#  stage: testing
#  only:
#    - main
#  script: |
#    #trigger the integration tests here

# This stage is triggered for main branch you set in the question,
# which will deploy the prod stage SAM application using
# the templated file generated.
deploy-prod:
  stage: prod
  # uncomment this to have a manual approval step before deployment to production
  # when: manual
  only:
    - main
  script:
    - . assume-role.sh ${PROD_PIPELINE_EXECUTION_ROLE} prod-deployment
    - sam deploy --stack-name ${PROD_STACK_NAME}
                 --template packaged-prod.yaml
                 --capabilities CAPABILITY_IAM
                 --region ${PROD_REGION}
                 --s3-bucket ${PROD_ARTIFACTS_BUCKET}
                 --no-fail-on-empty-changeset
                 --role-arn ${PROD_CLOUDFORMATION_EXECUTION_ROLE}
  • What troubleshooting steps have you already taken? Can you link to any docs or other resources so we know where you have been?
    • Attempted changing image to python:3.9, rather than the current value.

Thanks for taking the time to be thorough in your request, it really helps! :blush:

2

I am not familiar with sam, but it seems that the problem is not related to gitlab but to docker.

Docker cannot find the image hosted in ecr, it tries to fetch it locally and then from docker hub, it doesn’t know how to deal with ecr registry images.

samcli.local.docker.manager.DockerImagePullFailedException: Could not find public.ecr.aws/sam/build-python3.9:latest-x86_64 image locally and failed to pull it from docker.

Can you try to pull the image before starting sam ? This way it won’t try to pull it from docker hub.

- docker pull public.ecr.aws/sam/build-python3.9:latest-x86_64
- # your sam commands