Hi, so I’m wondering whether I’m not just not approaching the problem in the right way, or something else is missing. Basically I’d like to replace DinD with Kaniko within my CI pipeline running on EKS.
Although I have figured out how to build Dockerfile based jobs, I’m struggling with how to approach slightly more complex scenarios. In short, previously in a dind based stage I could execute arbitrary script commands in addition to building a container - whereas when using the kaniko image I am unable to do so.
My main questions currently are:
a) is it recommended practice to include supplementary script commands within the scope of a build phase?
b) is it possible to do something similar with a kaniko based workflow at all?
Any attempts I’ve made so far have resulted in failures, primarily due to the command binary e.g. apt, awscli, pip missing from the kaniko image itself (understandably so). e.g.
INFO[0021] Running: [/bin/sh -c apt-get update && apt-get install -y make]
/bin/sh: apt-get: command not found
error building image: error building stage: failed to execute command: waiting for process to exit: exit status 127
Thanks for replying. Yes the general requirement is to be able to create an image and push it to ECR (which kaniko can do) but then also execute additional steps such as updating and registering an ECS task definition within the context of a single build stage. e.g.
From my current understanding I would have to split the above into 2 separate stages. With the first using kaniko to build and the second, some other image that supports the required commands - the obvious downside being that additional time is required to download the second image.
Thanks for the context. That’s tricky, and after reading about Kaniko being able to specify --destination to push to an external registry, I’d say that a second CI/CD job might be the most boring solution here.
It may be tricky to build a new image that has all the dependencies pre-installed, using CI/CD and Kaniko, but it would be worth a shot.
Create a custom Dockerfile that uses a slim version of the Python 3.10 image (I guess that is the best bet since Python is a requirement). Docker
If there is caching needed, i.e. you want to keep the Kaniko generated image as binary blob around for the next CI/CD job, you can add a caching section like so;
FROM debian:bullseye
# the following line grabs /kaniko/executor from remote image
COPY --from=gcr.io/kaniko-project/executor:debug /kaniko/executor /kaniko/executor
CMD apt-get update && apt-get -y install jq curl python3 whatever you need && pip install awscli
As a positive side-effect, you’ll get all your toolchain ready in your build image so you don’t need to waste time and bandwidth with installing extra utilities in your before_script.
Oh cool, thanks a lot for sharing. I was unaware that you can “import” a binary from a remote container image in a Dockerfile. I think the binary architecture may be important to mention, compatible with debian:bullseye on x86_64 but likely won’t run on ARM-based architecture. Or am I mistaken here?
I tried for fun running GitLab EE which is AMD64 on macOS Ventura on Apple M1 which is ARM.
There is a warning in Docker Desktop that “Image may have poor performance or fail if run on (QEMU) emulation.”
There is also warning in GitLab log:
: (This is the expected behaviour if you are running under QEMU)
==> /var/log/gitlab/gitlab-rails/application.log <==
So AMD64 QEMU emulation is done by Docker Desktop for macOS automatically out of the box.
but it runs with performance not too bad
With respect to CPU architecture, the same limitations apply for COPY --from=someimage as for more common FROM someimage instruction.
The kaniko image is multiarch, so as far as you run docker build on amd64, arm64, or s390x you’re on the safe side.
Performance penalty of running qemulated amd64 images on arm64 arch is noticeable, but (usually) acceptable for regular developer experience. If possible, it’s always better to use image matching the docker host native architecture.
Thank you kindly @mouchar! your solution definitely hit the nail on the head.
Now to take things one step further I’ve been able to successfully integrate authentication to ECR as well.
Ensure that AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY exist as pipeline variables.
Based on my observations as long as these variables are present and your IAM policy is correctly configured, Kaniko is able to push to ecr without additional changes. This is in contrast to other posts I’ve encountered which seem to mention the creation of additional config files.
There’s still the matter of resolving the error the x509 error currently glossed over with --skip-tls-verify but as far as the original question is concerned - I think we’re done with it.
INFO[0000] Retrieving image manifest gcr.io/kaniko-project/executor:debug
INFO[0000] Retrieving image gcr.io/kaniko-project/executor:debug from registry gcr.io
error building image: Get "https://gcr.io/v2/": x509: certificate signed by unknown authority
As far as the x509 error is concerned, I don’t face this issue. gcr.io is signed by Google’s CA that is well-trusted by CA bundles of major operating systems. You’ll probably need to extend your apk add line with ca-certificates package to make any SSL-enabled app working correctly, without compromising security using --skip-tls-verify hacks
I think the binary architecture may be important to mention, compatible with 