I have an image built on master branch and another one that is built on branch checked out from master. I ran the build with script below
DOCKER_BUILDKIT=1 docker build \
--pull \
$CACHE_FROM_OPTIONS \
--build-arg BUILDKIT_INLINE_CACHE=1 \
--tag "$CI_REGISTRY_IMAGE:$CURRENT_BRANCH" \
--tag "$CI_REGISTRY_IMAGE:$APP_VERSION" \
--file "$DOCKERFILE" . && \
docker push "$CI_REGISTRY_IMAGE:$CURRENT_BRANCH" && \
(if [[ "$CURRENT_BRANCH" != "$APP_VERSION" ]]; then
docker push "$CI_REGISTRY_IMAGE:$APP_VERSION"
fi)
in $CACHE_FROM_OPTIONS example value looks something like this
--cache-from index.docker.io/myns/myapp:mytag
first lines of my Dockerfile are
FROM myns/php-base-docker-images:main
RUN install-php-extensions amqp-^1 imap bcmath gd
//some other stuff
My case is that image is build on master, inline cache is created, cache manifest is uploaded to registry.
Problem is that cache is not used for “RUN install-php-extensions amqp-^1 imap bcmath gd” in the build on my other branch. this RUN command does not change, php-base-docker-images image also changes rarely and I can verify if it changed or not. I can also see that image manifest is imported as docker build command prints that to the output.
Moreover, if the image is build and I run the build on master I am using --cache-from with tag from the source branch and then it uses the cache from image built on that branch. It is mind boggling, I run out of ideas.
What seems related is that even when I run docker build locally on my machine and I try to use --cache-from with the image build on gitlab I can see that cache is imported but not used. With no changes in the codebase and no COPY instruction that could (as I understand) invalidate the cache.
What can cause this ?
I have checked with dive 2 images, 1st built on master, 2nd built on branch. Last command from php base image has the same id/sha in both images but the next one, RUN with install-php-extensions has different identifiers. Both where built on gitlab runner and there is no difference in files being modified in that layer in both images.
on main image
Tags: (unavailable)
Id: 39a9676ad5d8baf25a03252675ee4c34b25611f3e8d1a53120bfc22a57be8c0b
Digest: sha256:2dcab550885b15f05a7290af363a6b815237a3a25465f9c0a9ed4a539980a2b7
Command:
RUN /bin/sh -c rm -rf docker # buildkit
Tags: (unavailable)
Id: 7ff3534f157543d1691e4c9117d2e2d4b26beb0160a3b139bafcc3f9c146d32e
Digest: sha256:f151959e42d1a19980a97b0da7d7a288cd8f2b0cccafcadebf69862f2aac2dc2
Command:
RUN /bin/sh -c install-php-extensions amqp-^1 imap bcmath gd # buildkit
on branch image
Tags: (unavailable)
Id: 39a9676ad5d8baf25a03252675ee4c34b25611f3e8d1a53120bfc22a57be8c0b
Digest: sha256:2dcab550885b15f05a7290af363a6b815237a3a25465f9c0a9ed4a539980a2b7
Command:
RUN /bin/sh -c rm -rf docker # buildkit
Tags: (unavailable)
Id: de86c06a7959398d2c86643e937c86811972d3c1f962e05d237ef9811364db3f
Digest: sha256:f3ef0af06f5c724a5cbc5690b24e43b61ac2bb651e7e60f693bbd4fd2b46c14d
Command:
RUN /bin/sh -c install-php-extensions amqp-^1 imap bcmath gd # buildkit
how the ids can differ ?
Another case with similar setup (master and fresh branch from master). This time docker used the cache and I can see that layers have same ids for that RUN command. Seems like something random, but since it can’t be random - what can be a reason for it ?
master image
Tags: (unavailable)
Id: 87dea064e45ab9329fd8f25b14de2ab932c86a545393843d606b12ad6803c482
Digest: sha256:5296dc6278d5bbd5068ec1ceeed0687bff2bc808d40eee301e6eec8aa7ea3e40
Command:
RUN /bin/sh -c install-php-extensions amqp-^1 imap bcmath gd # buildkit
branch image
Tags: (unavailable)
Id: 87dea064e45ab9329fd8f25b14de2ab932c86a545393843d606b12ad6803c482
Digest: sha256:5296dc6278d5bbd5068ec1ceeed0687bff2bc808d40eee301e6eec8aa7ea3e40
Command:
RUN /bin/sh -c install-php-extensions amqp-^1 imap bcmath gd # buildkit
I banged my head against the wall, until I got this working and the required switch is --cache-to type=inline, only images created with that switch can be used as cache for --cache-from.