Hi, I’m trying to do a multi-stage docker build in GitLab-CI, but one of the layers in the docker cache gets invalidated when changing parts of the source code. That should not happen or at least it doesn’t happen when I build the Dockerfile on my computer.
My final goal is to make use of cargo chef, a tool that allows one to build the dependencies of a rust project independently of the rest of the code. Cargo chef comes with two flags: cargo chef --prepare and cargo chef --cook. cargo chef --prepare creates a recipe.json file that stores all the names and versions of the dependencies. cargo chef --cook builds the dependencies from a *recipe.json" file. The idea is to put cargo chef --prepare and cargo chef --cook in different (docker) build stages (“planner” stage and “builder” stage, resp.) and to only copy the “recipe.json” from the planner stage to the builder stage. If some part of the docker context is changed, the planner stage might be build from scratch, but as long as the recipe.json stays the same, the cache can be used in the builder stage again.
I’m following this guide https://www.lpalmieri.com/posts/fast-rust-docker-builds/ and everything is much better explained there.
The issue appears if I make a small change to my source code. Adding a simple println!("123"); to my main.rs makes the builder stage forget about the cache and build everything from scratch - even though the “recipe.json” didn’t change. Throughout all my testing I made sure that the actual content of the “recipe.json” didn’t change by tracking the sha256 hash.
Output from the GitLab-CI pipeline:
...
$ docker build --target planner --cache-from $CI_REGISTRY_IMAGE:planner --tag $CI_REGISTRY_IMAGE:planner --build-arg BUILDKIT_INLINE_CACHE=1 .
#0 building with "default" instance using docker driver
#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 719B done
#1 DONE 0.0s
#2 [internal] load .dockerignore
#2 transferring context: 124B done
#2 DONE 0.0s
#3 [internal] load metadata for docker.io/lukemathwalker/cargo-chef:latest
#3 DONE 0.2s
#4 [chef 1/2] FROM docker.io/lukemathwalker/cargo-chef:latest@sha256:f62bd9569ea545af4ff203d18a0c51efd38129aedc3ceee1c3a2ab769af9a281
#4 DONE 0.0s
#5 importing cache manifest from [redacted]:planner
#5 DONE 0.0s
#6 [internal] load build context
#6 transferring context: 11.39kB done
#6 DONE 0.0s
#7 [chef 2/2] WORKDIR /app
#7 CACHED
#8 [planner 1/2] COPY . .
#8 DONE 0.0s
#9 [planner 2/2] RUN cargo chef prepare --recipe-path recipe.json
#9 DONE 0.3s
#10 preparing layers for inline cache
#10 DONE 1.0s
#11 exporting to image
#11 exporting layers done
#11 writing image sha256:656d0121c2882df3bb980622a9a1985d23e9e3d15b2674bb3641e1570bf9af69 done
#11 naming to [redacted]:planner done
#11 DONE 0.0s
$ docker build --target builder --cache-from $CI_REGISTRY_IMAGE:builder --tag $CI_REGISTRY_IMAGE:builder --build-arg BUILDKIT_INLINE_CACHE=1 .
#0 building with "default" instance using docker driver
#1 [internal] load .dockerignore
#1 transferring context: 124B done
#1 DONE 0.0s
#2 [internal] load build definition from Dockerfile
#2 transferring dockerfile: 719B done
#2 DONE 0.0s
#3 [internal] load metadata for docker.io/lukemathwalker/cargo-chef:latest
#3 DONE 0.2s
#4 [chef 1/2] FROM docker.io/lukemathwalker/cargo-chef:latest@sha256:f62bd9569ea545af4ff203d18a0c51efd38129aedc3ceee1c3a2ab769af9a281
#4 DONE 0.0s
#5 importing cache manifest from [redacted]:builder
#5 DONE 0.0s
#6 [internal] load build context
#6 transferring context: 144B done
#6 DONE 0.0s
#7 [planner 1/2] COPY . .
#7 CACHED
#8 [planner 2/2] RUN cargo chef prepare --recipe-path recipe.json
#8 CACHED
#9 [chef 2/2] WORKDIR /app
#9 CACHED
#10 [builder 1/4] COPY --from=planner /app/recipe.json recipe.json
#10 DONE 0.0s
#11 [builder 2/4] RUN cargo chef cook --release --recipe-path recipe.json
#11 0.290 Updating crates.io index
#11 0.531 Downloading crates ...
#11 0.862 Downloaded parking_lot v0.12.1
#11 0.876 Downloaded mio v0.8.10
#11 0.885 Downloaded parking_lot_core v0.9.9
#11 0.888 Downloaded smallvec v1.11.2
#11 0.903 Downloaded num_cpus v1.16.0
#11 0.939 Downloaded syn v2.0.41
#11 0.954 Downloaded tokio-macros v2.2.0
#11 0.955 Downloaded signal-hook-registry v1.4.1
#11 0.956 Downloaded proc-macro2 v1.0.70
#11 0.960 Downloaded cfg-if v1.0.0
#11 0.966 Downloaded pin-project-lite v0.2.13
#11 0.973 Downloaded socket2 v0.5.5
#11 0.976 Downloaded unicode-ident v1.0.12
#11 0.980 Downloaded scopeguard v1.2.0
#11 0.981 Downloaded quote v1.0.33
#11 0.991 Downloaded bytes v1.5.0
#11 1.078 Downloaded lock_api v0.4.11
#11 1.090 Downloaded autocfg v1.1.0
#11 1.178 Downloaded libc v0.2.151
#11 1.221 Downloaded tokio v1.35.1
#11 1.290 Compiling libc v0.2.151
#11 1.290 Compiling proc-macro2 v1.0.70
#11 1.290 Compiling unicode-ident v1.0.12
#11 1.290 Compiling autocfg v1.1.0
#11 1.291 Compiling parking_lot_core v0.9.9
#11 1.292 Compiling cfg-if v1.0.0
#11 1.292 Compiling scopeguard v1.2.0
#11 1.292 Compiling smallvec v1.11.2
#11 1.293 Compiling bytes v1.5.0
#11 1.293 Compiling pin-project-lite v0.2.13
#11 1.540 Compiling lock_api v0.4.11
#11 2.013 Compiling quote v1.0.33
#11 2.158 Compiling syn v2.0.41
#11 2.262 Compiling mio v0.8.10
#11 2.262 Compiling num_cpus v1.16.0
#11 2.262 Compiling signal-hook-registry v1.4.1
#11 2.262 Compiling socket2 v0.5.5
#11 2.576 Compiling parking_lot v0.12.1
#11 5.189 Compiling tokio-macros v2.2.0
#11 5.752 Compiling tokio v1.35.1
#11 12.11 Compiling hello-world v0.0.1 (/app)
#11 12.27 Finished release [optimized] target(s) in 12.00s
#11 DONE 12.4s
#12 [builder 3/4] COPY . .
#12 DONE 0.0s
#13 [builder 4/4] RUN cargo build --release --bin hello-world
#13 0.381 Compiling hello-world v0.1.0 (/app)
#13 0.844 Finished release [optimized] target(s) in 0.49s
#13 DONE 0.9s
#14 preparing layers for inline cache
#14 DONE 1.5s
#15 exporting to image
#15 exporting layers done
#15 writing image sha256:e0ea0e7438b9e12a1812df29bd72356f760308a766bd1e15a3c792126e10d09d done
#15 naming to [redacted]:builder done
#15 DONE 0.0s
What I would actually expect to see and what I see locally on my machine:
*$ docker build --target planner --cache-from $CI_REGISTRY_IMAGE:planner --tag $CI_REGISTRY_IMAGE:planner --build-arg BUILDKIT_INLINE_CACHE=1 . [INSERTED, not part of the actual output]*
#0 building with "default" instance using docker driver
#1 [internal] load .dockerignore
#1 transferring context: 124B done
#1 DONE 0.0s
#2 [internal] load build definition from Dockerfile
#2 transferring dockerfile: 518B done
#2 DONE 0.0s
#3 [internal] load metadata for docker.io/lukemathwalker/cargo-chef:latest
#3 DONE 0.2s
#4 importing cache manifest from [redacted]:planner
#4 DONE 0.0s
#5 [chef 1/2] FROM docker.io/lukemathwalker/cargo-chef:latest@sha256:f62bd9569ea545af4ff203d18a0c51efd38129aedc3ceee1c3a2ab769af9a281
#5 DONE 0.0s
#6 [internal] load build context
#6 transferring context: 1.50kB done
#6 DONE 0.0s
#7 [chef 2/2] WORKDIR /app
#7 CACHED
#8 [planner 1/2] COPY . .
#8 DONE 0.0s
#9 [planner 2/2] RUN cargo chef prepare --recipe-path recipe.json
#9 DONE 0.4s
#10 preparing layers for inline cache
#10 DONE 0.8s
#11 exporting to image
#11 exporting layers done
#11 writing image sha256:2c823018ad126d4722eba54c9faa3a3eceb29cb770031cba9ac01d268a9d44f7 done
#11 naming to [redacted]:planner done
#11 DONE 0.0s
*$ docker build --target builder --cache-from $CI_REGISTRY_IMAGE:builder --tag $CI_REGISTRY_IMAGE:builder --build-arg BUILDKIT_INLINE_CACHE=1 . [INSERTED, not part of the actual output]*
#0 building with "default" instance using docker driver
#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 518B done
#1 DONE 0.0s
#2 [internal] load .dockerignore
#2 transferring context: 124B done
#2 DONE 0.0s
#3 [internal] load metadata for docker.io/lukemathwalker/cargo-chef:latest
#3 DONE 0.2s
#4 [chef 1/2] FROM docker.io/lukemathwalker/cargo-chef:latest@sha256:f62bd9569ea545af4ff203d18a0c51efd38129aedc3ceee1c3a2ab769af9a281
#4 DONE 0.0s
#5 importing cache manifest from [redacted]:builder
#5 DONE 0.0s
#6 [internal] load build context
#6 transferring context: 144B done
#6 DONE 0.0s
#7 [planner 1/2] COPY . .
#7 CACHED
#8 [chef 2/2] WORKDIR /app
#8 CACHED
#9 [planner 2/2] RUN cargo chef prepare --recipe-path recipe.json
#9 CACHED
#10 [builder 1/4] COPY --from=planner /app/recipe.json recipe.json
#10 CACHED
#11 [builder 2/4] RUN cargo chef cook --release --recipe-path recipe.json
#11 CACHED
#12 [builder 3/4] COPY . .
#12 DONE 0.0s
#13 [builder 4/4] RUN cargo build --release --bin hello-world
#13 0.338 Compiling hello-world v0.1.0 (/app)
#13 0.878 Finished release [optimized] target(s) in 0.58s
#13 DONE 0.9s
#14 preparing layers for inline cache
#14 DONE 0.8s
#15 exporting to image
#15 exporting layers done
#15 writing image sha256:a994ea4f6331f437ff1b1ad38002b8c52731fdabfd4c7c9c487714003242cbe1 done
#15 naming to [redacted]:builder done
#15 DONE 0.0s
Copying of the context and computing of the “recipe.json” is done in both cases (in GitLab-CI and on my machine):
#8 [planner 1/2] COPY . .
#8 DONE 0.0s
#9 [planner 2/2] RUN cargo chef prepare --recipe-path recipe.json
#9 DONE 0.4s
I think this part is how it’s supposed to be.
The copying of the “recipe.json” is only done in GitLab-CI though:
In GitLab-CI:
...
#10 [builder 1/4] COPY --from=planner /app/recipe.json recipe.json
#10 DONE 0.0s
#11 [builder 2/4] RUN cargo chef cook --release --recipe-path recipe.json
#11 0.290 Updating crates.io index
...
vs.
Locally:
#10 [builder 1/4] COPY --from=planner /app/recipe.json recipe.json
#10 CACHED
#11 [builder 2/4] RUN cargo chef cook --release --recipe-path recipe.json
#11 CACHED
Dockerfile:
# Leveraging the pre-built Docker images with
# cargo-chef and the Rust toolchain
FROM lukemathwalker/cargo-chef:latest AS chef
WORKDIR /app
FROM chef AS planner
COPY . .
RUN cargo chef prepare --recipe-path recipe.json
FROM chef AS builder
COPY --from=planner /app/recipe.json recipe.json
# Build dependencies - this is the caching Docker layer!
RUN cargo chef cook --release --recipe-path recipe.json
# Build application
COPY . .
RUN cargo build --release --bin hello-world
.gitlab-ci.yml:
variables:
DOCKER_BUILDKIT: 1
DOCKER_DIND: "24.0.7-dind"
DOCKER_TLS_CERTDIR: ""
DOCKER_HOST: "tcp://docker:2375"
FF_NETWORK_PER_BUILD: 1
stages:
- test
test-image:
image: docker:latest
stage: test
services:
- name: docker:dind
alias: docker
command: ["--tls=false", "--insecure-registry=[redacted]"]
script:
- echo $DOCKER_BUILDKIT
- echo $CI_REGISTRY_IMAGE
- echo $CI_REGISTRY
- docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
- docker pull $CI_REGISTRY_IMAGE:chef || true
- docker pull $CI_REGISTRY_IMAGE:planner || true
- docker pull $CI_REGISTRY_IMAGE:builder || true
- docker build
--target chef
--cache-from $CI_REGISTRY_IMAGE:chef
--tag $CI_REGISTRY_IMAGE:chef
--build-arg BUILDKIT_INLINE_CACHE=1
.
- docker build
--target planner
--cache-from $CI_REGISTRY_IMAGE:planner
--tag $CI_REGISTRY_IMAGE:planner
--build-arg BUILDKIT_INLINE_CACHE=1
.
- docker build
--target builder
--cache-from $CI_REGISTRY_IMAGE:builder
--tag $CI_REGISTRY_IMAGE:builder
--build-arg BUILDKIT_INLINE_CACHE=1
.
- docker push $CI_REGISTRY_IMAGE:chef
- docker push $CI_REGISTRY_IMAGE:planner
- docker push $CI_REGISTRY_IMAGE:builder
- docker run
--network=host $CI_REGISTRY_IMAGE:builder
I’m using Docker version 24.0.7 locally and a self-managed GitLab Enterprise Edition v16.5.1-ee.