Kaniko custom builds

Hi, so I’m wondering whether I’m not just not approaching the problem in the right way, or something else is missing. Basically I’d like to replace DinD with Kaniko within my CI pipeline running on EKS.
Although I have figured out how to build Dockerfile based jobs, I’m struggling with how to approach slightly more complex scenarios. In short, previously in a dind based stage I could execute arbitrary script commands in addition to building a container - whereas when using the kaniko image I am unable to do so.

DinD based example:

image: docker:dind
services:
  - docker:18-dind
before_script:
  - apk add --no-cache curl jq python3 py3-pip
  - pip install awscli
script:
  - docker build -f docker/Dockerfile .
  - <arbitrary awscli command>

kaniko based example (presently missing functionality outside of building)

image:
  name: gcr.io/kaniko-project/executor:debug
  entrypoint: [""]
script:
  - /kaniko/executor
    --context "${CI_PROJECT_DIR}"
    --dockerfile "docker/Dockerfile"
    --no-push

My main questions currently are:
a) is it recommended practice to include supplementary script commands within the scope of a build phase?
b) is it possible to do something similar with a kaniko based workflow at all?

Any attempts I’ve made so far have resulted in failures, primarily due to the command binary e.g. apt, awscli, pip missing from the kaniko image itself (understandably so). e.g.

INFO[0021] Running: [/bin/sh -c apt-get update && apt-get install -y make] 
/bin/sh: apt-get: command not found
error building image: error building stage: failed to execute command: waiting for process to exit: exit status 127

Any advice would be highly appreciated.

1 Like

I haven’t tried it, but the Kaniko container image is not Debian/Ubuntu based, but FROM scratch. kaniko/Dockerfile_slim at fe2413e6e3c8caf943d50cf1d233a561943df1d6 · GoogleContainerTools/kaniko · GitHub meaning that there is no package manager or full Linux distribution available in this context.

I could execute arbitrary script commands in addition to building a container

Is the requirement to test the built container image using awscli somehow? I’m not sure I understand it correctly.

1 Like

Thanks for replying. Yes the general requirement is to be able to create an image and push it to ECR (which kaniko can do) but then also execute additional steps such as updating and registering an ECS task definition within the context of a single build stage. e.g.

script:
  - docker build -f docker/Dockerfile .
  - envsubst < task-template.json > new-task.json
  - aws ecs register-task-definition --cli-input-json file://new-task.json 

From my current understanding I would have to split the above into 2 separate stages. With the first using kaniko to build and the second, some other image that supports the required commands - the obvious downside being that additional time is required to download the second image.

1 Like

Thanks for the context. That’s tricky, and after reading about Kaniko being able to specify --destination to push to an external registry, I’d say that a second CI/CD job might be the most boring solution here.

It may be tricky to build a new image that has all the dependencies pre-installed, using CI/CD and Kaniko, but it would be worth a shot.

  1. Create a custom Dockerfile that uses a slim version of the Python 3.10 image (I guess that is the best bet since Python is a requirement). Docker
  2. Install curl, jq, and awscli via pip
  3. Push the image to the GitLab container registry
  4. Specify the image keyword for the upload job

Untested, something like this as Dockerfile.

FROM python:3.11-slim-bullseye

CMD apt update && apt -y install jq curl && pip install awscli 

and CI/CD config

kaniko-build: 
  image:
    name: gcr.io/kaniko-project/executor:debug
    entrypoint: [""]
  script:
    - /kaniko/executor
      --context "${CI_PROJECT_DIR}"
      --dockerfile "docker/Dockerfile"
      --destination <aws ecr registry here>

post-steps:
  image: registry.gitlab.com/path/to/custom/python/image 
  script: 
     - awscli <do something> 

If there is caching needed, i.e. you want to keep the Kaniko generated image as binary blob around for the next CI/CD job, you can add a caching section like so;

cache:
  key: $CI_COMMIT_REF_SLUG
  paths:
    - <kaniko-output-dir>

Or you may extend your build image with the kaniko executor binary and run all your job commands within a single job.

Dockerfile of build image (registry.gitlab.com/yourname/yourproject:latest):

FROM debian:bullseye
# the following line grabs /kaniko/executor from remote image
COPY --from=gcr.io/kaniko-project/executor:debug /kaniko/executor /kaniko/executor

CMD apt-get update && apt-get -y install jq curl python3 whatever you need && pip install awscli

Job:

your-job-name:
  image: registry.gitlab.com/yourname/yourproject:latest
  script:
  - /kaniko/executor
    --context "${CI_PROJECT_DIR}"
    --dockerfile "docker/Dockerfile"
    --destination=yourregistry/namespace/name:${CI_COMMIT_SHORT_SHA}
  - envsubst < task-template.json > new-task.json
  - aws ecs register-task-definition --cli-input-json file://new-task.json

As a positive side-effect, you’ll get all your toolchain ready in your build image so you don’t need to waste time and bandwidth with installing extra utilities in your before_script.

2 Likes

Oh cool, thanks a lot for sharing. I was unaware that you can “import” a binary from a remote container image in a Dockerfile. :slight_smile: I think the binary architecture may be important to mention, compatible with debian:bullseye on x86_64 but likely won’t run on ARM-based architecture. Or am I mistaken here?

Hi dnsmichi,

…x86_64 architecture running on ARM based…

I tried for fun running GitLab EE which is AMD64 on macOS Ventura on Apple M1 which is ARM.
There is a warning in Docker Desktop that “Image may have poor performance or fail if run on (QEMU) emulation.”
There is also warning in GitLab log:
: (This is the expected behaviour if you are running under QEMU)
==> /var/log/gitlab/gitlab-rails/application.log <==

So AMD64 QEMU emulation is done by Docker Desktop for macOS automatically out of the box.
but it runs with performance not too bad

With respect to CPU architecture, the same limitations apply for COPY --from=someimage as for more common FROM someimage instruction.

The kaniko image is multiarch, so as far as you run docker build on amd64, arm64, or s390x you’re on the safe side.

Performance penalty of running qemulated amd64 images on arm64 arch is noticeable, but (usually) acceptable for regular developer experience. If possible, it’s always better to use image matching the docker host native architecture.

1 Like

Makes sense, thanks a lot for clarifying! :slight_smile:

Thank you kindly @mouchar! your solution definitely hit the nail on the head.
Now to take things one step further I’ve been able to successfully integrate authentication to ECR as well.

Ensure that AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY exist as pipeline variables.
Based on my observations as long as these variables are present and your IAM policy is correctly configured, Kaniko is able to push to ecr without additional changes. This is in contrast to other posts I’ve encountered which seem to mention the creation of additional config files.

Dockerfile for the proposed base image (builder)

FROM alpine
RUN apk add --no-cache jq curl python3 py3-pip gettext libintl && pip install awscli
COPY --from=gcr.io/kaniko-project/executor:debug /kaniko/executor /kaniko/executor

.gitlab-ci.yml using pushed builder

variables:
  BUILDER_IMAGE: "<accountid>.dkr.ecr.<region>.amazonaws.com/<repo>"
  TARGET_IMAGE: "<accountid>.dkr.ecr.<region>.amazonaws.com/<repo2>"

# build and push simple image
build:kaniko-builder:
  image: $BUILDER_IMAGE:latest
  script:
  - /kaniko/executor
    --context $CI_PROJECT_DIR
    --dockerfile "docker/Dockerfile"
    --destination $TARGET_IMAGE:latest
    --destination $TARGET_IMAGE:$CI_BUILD_REF
    --skip-tls-verify
  - aws sts get-caller-identity #test command

There’s still the matter of resolving the error the x509 error currently glossed over with --skip-tls-verify but as far as the original question is concerned - I think we’re done with it.

INFO[0000] Retrieving image manifest gcr.io/kaniko-project/executor:debug
INFO[0000] Retrieving image gcr.io/kaniko-project/executor:debug from registry gcr.io
error building image: Get "https://gcr.io/v2/": x509: certificate signed by unknown authority
1 Like

Hi, I’m glad to help.

As far as the x509 error is concerned, I don’t face this issue. gcr.io is signed by Google’s CA that is well-trusted by CA bundles of major operating systems. You’ll probably need to extend your apk add line with ca-certificates package to make any SSL-enabled app working correctly, without compromising security using --skip-tls-verify hacks :wink:

2 Likes