Docker build not using cache

I am trying to speed up my Docker build by using the --cache-from option.
This works on my local machine but not on Gitlab (EE) and I ran out of ideas. Can anyone please take a look and maybe see what I am missing?

This is my Dockerfile, notice that it consists of multiple stages:

# Build application
FROM node:12 AS builder
WORKDIR /app
COPY package*.json ./
RUN npm install
COPY . .
RUN npm run build

# Deploy on NGINX
FROM nginx:alpine
COPY ./nginx/nginx.conf.template /etc/nginx/templates/default.conf.template
COPY ./nginx/envsubst-on-index.html.sh /docker-entrypoint.d/00-envsubst-on-index.html.sh
RUN chmod +x /docker-entrypoint.d/00-envsubst-on-index.html.sh
WORKDIR /usr/share/nginx/html
RUN rm -rf ./*
COPY --from=builder /app/build .

Now if I run the commands below locally using the latest image stored in the Gitlab registry the build is incredibly fast since everything is cached.
Notice that I have purposely removed all local images and logged in to my Gitlab registry.

PS D:\temp\gitlab-workflow> docker image ls -a
REPOSITORY   TAG       IMAGE ID   CREATED   SIZE
PS D:\temp\gitlab-workflow> docker login gitlab.example.com:5050
Authenticating with existing credentials...
Login Succeeded
PS D:\temp\gitlab-workflow> docker --version
Docker version 20.10.5, build 55c4c88
PS D:\temp\gitlab-workflow>
PS D:\temp\gitlab-workflow>
PS D:\temp\gitlab-workflow> docker build --build-arg BUILDKIT_INLINE_CACHE=1 --cache-from gitlab.example.com:5050/petermarcoen/gitlab-workflow:latest -t gitlab.example.com:5050/petermarcoen/gitlab-workflow:latest .
[+] Building 27.7s (24/24) FINISHED
 => [internal] load build definition from Dockerfile                                                                                                                                                 0.0s 
 => => transferring dockerfile: 518B                                                                                                                                                                 0.0s 
 => [internal] load .dockerignore                                                                                                                                                                    0.0s 
 => => transferring context: 2B                                                                                                                                                                      0.0s 
 => [internal] load metadata for docker.io/library/nginx:alpine                                                                                                                                      1.7s 
 => [internal] load metadata for docker.io/library/node:12                                                                                                                                           1.7s 
 => [auth] library/nginx:pull token for registry-1.docker.io                                                                                                                                         0.0s 
 => [auth] library/node:pull token for registry-1.docker.io                                                                                                                                          0.0s 
 => importing cache manifest from gitlab.example.com:5050/petermarcoen/gitlab-workflow:latest                                                                                                          1.1s 
 => [stage-1 1/7] FROM docker.io/library/nginx:alpine@sha256:07ab71a2c8e4ecb19a5a5abcfb3a4f175946c001c8af288b1aa766d67b0d05d2                                                                        0.0s 
 => [builder 1/6] FROM docker.io/library/node:12@sha256:609103746810535f5a3a987a26ba4ce95d96225d28e9d6228faa5aa331980f37                                                                             0.0s 
 => [internal] load build context                                                                                                                                                                    4.2s 
 => [auth] petermarcoen/gitlab-workflow:pull token for gitlab.example.com:5050                                                                                                                         0.0s 
 => CACHED [builder 2/6] WORKDIR /app                                                                                                                                                                0.0s 
 => CACHED [builder 3/6] COPY package*.json ./                                                                                                                                                       0.0s 
 => CACHED [builder 4/6] RUN npm install                                                                                                                                                             0.0s 
 => [builder 5/6] COPY . .                                                                                                                                                                           6.8s 
 => [builder 6/6] RUN npm run build                                                                                                                                                                 12.8s 
 => CACHED [stage-1 2/7] COPY ./nginx/nginx.conf.template /etc/nginx/templates/default.conf.template                                                                                                 0.0s 
 => CACHED [stage-1 3/7] COPY ./nginx/envsubst-on-index.html.sh /docker-entrypoint.d/00-envsubst-on-index.html.sh                                                                                    0.0s 
 => CACHED [stage-1 4/7] RUN chmod +x /docker-entrypoint.d/00-envsubst-on-index.html.sh                                                                                                              0.0s 
 => CACHED [stage-1 5/7] WORKDIR /usr/share/nginx/html                                                                                                                                               0.0s 
 => CACHED [stage-1 6/7] RUN rm -rf ./*                                                                                                                                                              0.0s 
 => [stage-1 7/7] COPY --from=builder /app/build .                                                                                                                                                   0.1s 
 => exporting to image                                                                                                                                                                               0.0s 
 => => exporting layers                                                                                                                                                                              0.0s 
 => => writing image sha256:48e4c21e9ad8377376041055fc6c184f64c0c531c0171c4d4a5a027497f9c0bf                                                                                                         0.0s 
 => => naming to gitlab.example.com:5050/petermarcoen/gitlab-workflow:latest                                                                                                                           0.0s 
 => exporting cache                                                                                                                                                                                  0.0s 
 => => preparing build cache for export                                                                                                                                                              0.0s 

Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them
PS D:\temp\gitlab-workflow>

Notice that for example step [builder 4/6] is CACHED.
In my gitlab-ci.yml I (think I) do the same thing:

variables:
  DOCKER_BUILDKIT: 1
stages:
  - build
build:
  image: docker:20.10.5
  stage: build
  tags:
    - docker
  services:
    - name: docker:stable-dind
  script:
    - echo "$CI_REGISTRY_IMAGE:latest"
    - docker --version
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
    - docker pull $CI_REGISTRY_IMAGE:latest || true # Pull latest image to leverage layer caching, true to avoid failing if image is not found
    - docker build --build-arg BUILDKIT_INLINE_CACHE=1 --cache-from $CI_REGISTRY_IMAGE:latest -t $CI_REGISTRY_IMAGE:latest . # Create an image tagged latest
    - docker push $CI_REGISTRY_IMAGE:latest # Push the image to the Gitlab registry

But here the cache is not used and everything is done from scratch:

Running with gitlab-runner 13.10.0 (54944146)
  on master-3 YT2nPraF
section_start:1619689332:prepare_executor
Preparing the "docker" executor
Using Docker executor with image docker:20.10.5 ...
Starting service docker:stable-dind ...
Pulling docker image docker:stable-dind ...
Using docker image sha256:6f19136cf89d49aee8fa560eaab20e034c6ee17bbce55a9d33e1fbe5baa35b03 for docker:stable-dind with digest docker@sha256:4972457c6a8a4309f9796fc3c8fd288923045ba0e214c102d92b70be99e249d1 ...
Waiting for services to be up and running...
Pulling docker image docker:20.10.5 ...
Using docker image sha256:1588477122de4fdfe9fcb9ddeeee6ac6b93e9e05a65c68a6e22add0a98b8e0fe for docker:20.10.5 with digest docker@sha256:7ed427295687586039ff3433bb9b4419c5cf1e6294025dadf7641126665a78f5 ...
section_end:1619689348:prepare_executor
section_start:1619689348:prepare_script
Preparing environment
Running on runner-yt2npraf-project-4-concurrent-0 via c2c25f0e33b6...
section_end:1619689348:prepare_script
section_start:1619689348:get_sources
Getting source from Git repository
Fetching changes with git depth set to 50...
Reinitialized existing Git repository in /builds/petermarcoen/gitlab-workflow/.git/
Checking out b6f85f62 as master...

Skipping Git submodules setup
section_end:1619689350:get_sources
section_start:1619689350:step_script
Executing "step_script" stage of the job script
Using docker image sha256:1588477122de4fdfe9fcb9ddeeee6ac6b93e9e05a65c68a6e22add0a98b8e0fe for docker:20.10.5 with digest docker@sha256:7ed427295687586039ff3433bb9b4419c5cf1e6294025dadf7641126665a78f5 ...
$ echo "$CI_REGISTRY_IMAGE:latest"
gitlab.example.com:5050/petermarcoen/gitlab-workflow:latest
$ docker --version
Docker version 20.10.5, build 55c4c88
$ docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded
$ docker pull $CI_REGISTRY_IMAGE:latest || true
latest: Pulling from petermarcoen/gitlab-workflow
540db60ca938: Pulling fs layer
...
f8277735e4e6: Pull complete
Digest: sha256:9866109bf8d5e0bb7464c296d26894656fc5ca05010363968db2b7b20e21ab91
Status: Downloaded newer image for gitlab.example.com:5050/petermarcoen/gitlab-workflow:latest
gitlab.example.com:5050/petermarcoen/gitlab-workflow:latest
$ docker build --build-arg BUILDKIT_INLINE_CACHE=1 --cache-from $CI_REGISTRY_IMAGE:latest -t $CI_REGISTRY_IMAGE:latest .
#1 [internal] load .dockerignore
#1 sha256:e60682f2aa9fc4906b9bada9c7560194e970b39baa2b520526f4bd88285e81b4
#1 transferring context: 2B done
#1 DONE 0.1s

#2 [internal] load build definition from Dockerfile
#2 sha256:7c337464965d70e2254ba0a93736318dd78dd915600cf36665260af68bf62844
#2 transferring dockerfile: 503B done
#2 DONE 0.1s

#4 [internal] load metadata for docker.io/library/node:12
#4 sha256:55906eb64e7bc4d69cef5ad3f3679feeb2af997de97f0dab4394b48f2c72efd2
#4 DONE 1.6s

#3 [internal] load metadata for docker.io/library/nginx:alpine
#3 sha256:b001d263a254f0e4960d52c837d5764774ef80ad3878c61304052afb6e0e9af2
#3 DONE 1.8s

#19 importing cache manifest from gitlab.example.com:5050/petermarcoen/gitlab-workflow:latest
#19 sha256:cb3fd8bb86fa9e8874fecf3ea295257e0ebcb4b1eb5cf3d6d3e7a515569cb16c
#19 DONE 0.0s

#5 [stage-1 1/7] FROM docker.io/library/nginx:alpine@sha256:07ab71a2c8e4ecb19a5a5abcfb3a4f175946c001c8af288b1aa766d67b0d05d2
#5 sha256:9e7cf3b35eef66df544c5643e6f1964f9ed0e9beae5c6a12bd7763d532039238
#5 DONE 0.0s

#6 [internal] load build context
#6 sha256:2a6388e58f30fc93807b1f1a4797c36879dc0cff5ac162a11e608a6a8c6b5804
#6 transferring context: 2.11MB 0.1s done
#6 DONE 0.1s

#12 [builder 1/6] FROM docker.io/library/node:12@sha256:609103746810535f5a3a987a26ba4ce95d96225d28e9d6228faa5aa331980f37
#12 sha256:049f879149ced3f210db72f418337b07e9e993775dca36fd9a970b248c8fec14
#12 resolve docker.io/library/node:12@sha256:609103746810535f5a3a987a26ba4ce95d96225d28e9d6228faa5aa331980f37 done
#12 sha256:3ff9a21f91c215efa3b65ae958df9bd5011e55301849ba0789e2788575b1c0df 2.21kB / 2.21kB done
...
#12 extracting sha256:20f8d3e06d470f5659ac84232f4d5b309ad360a0b87a01f8042a020296616d69 done
#12 DONE 44.0s

#13 [builder 2/6] WORKDIR /app
#13 sha256:7bd1b19391b06eddb9e1b12d94cf7c192f6ef4798bc1e7315b4117bba5103b72
#13 DONE 24.6s

#14 [builder 3/6] COPY package*.json ./
#14 sha256:60a7b5046cab58d2e81a646151e00b19dc1074bccfef323f727cd15fb816caa8
#14 DONE 0.1s

#15 [builder 4/6] RUN npm install
#15 sha256:ffda396506740db21ca2da5b5ac875fc2cee6c670777216d32490345c62cb8fa
#15 49.58 
#15 49.58 > core-js@2.6.12 postinstall /app/node_modules/babel-runtime/node_modules/core-js
#15 49.58 > node -e "try{require('./postinstall')}catch(e){}"
#15 49.58 
#15 49.71 Thank you for using core-js ( https://github.com/zloirock/core-js ) for polyfilling JavaScript standard library!
#15 49.71 
#15 49.71 The project needs your help! Please consider supporting of core-js on Open Collective or Patreon: 
#15 49.71 > https://opencollective.com/core-js 
#15 49.71 > https://www.patreon.com/zloirock 
#15 49.71 
#15 49.71 Also, the author of core-js ( https://github.com/zloirock ) is looking for a good job -)
#15 49.71 
#15 49.84 
#15 49.84 > core-js@3.10.2 postinstall /app/node_modules/core-js
#15 49.84 > node -e "try{require('./postinstall')}catch(e){}"
#15 49.84 
#15 49.91 Thank you for using core-js ( https://github.com/zloirock/core-js ) for polyfilling JavaScript standard library!
#15 49.91 
#15 49.91 The project needs your help! Please consider supporting of core-js on Open Collective or Patreon: 
#15 49.91 > https://opencollective.com/core-js 
#15 49.91 > https://www.patreon.com/zloirock 
#15 49.91 
#15 49.91 Also, the author of core-js ( https://github.com/zloirock ) is looking for a good job -)
#15 49.91 
#15 49.96 
#15 49.96 > core-js-pure@3.10.2 postinstall /app/node_modules/core-js-pure
#15 49.96 > node -e "try{require('./postinstall')}catch(e){}"
#15 49.96 
#15 50.15 Thank you for using core-js ( https://github.com/zloirock/core-js ) for polyfilling JavaScript standard library!
#15 50.15 
#15 50.15 The project needs your help! Please consider supporting of core-js on Open Collective or Patreon: 
#15 50.15 > https://opencollective.com/core-js 
#15 50.15 > https://www.patreon.com/zloirock 
#15 50.15 
#15 50.15 Also, the author of core-js ( https://github.com/zloirock ) is looking for a good job -)
#15 50.15 
#15 50.22 
#15 50.22 > ejs@2.7.4 postinstall /app/node_modules/ejs
#15 50.22 > node ./postinstall.js
#15 50.22 
#15 50.33 Thank you for installing e[35mEJS: built with the e[32mJake JavaScript build tool (e[32mhttps://jakejs.com/)
#15 50.33 
#15 53.32 npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@1.2.13 (node_modules/webpack-dev-server/node_modules/fsevents):
#15 53.32 npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@1.2.13: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})
#15 53.34 npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@1.2.13 (node_modules/watchpack-chokidar2/node_modules/fsevents):
#15 53.34 npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@1.2.13: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})
#15 53.35 npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@2.3.2 (node_modules/fsevents):
#15 53.35 npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@2.3.2: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})
#15 53.35 
#15 53.36 added 1949 packages from 765 contributors and audited 1952 packages in 52.535s
#15 55.11 
#15 55.11 135 packages are looking for funding
#15 55.11   run `npm fund` for details
#15 55.11 
#15 55.11 found 0 vulnerabilities
#15 55.11 
#15 DONE 56.7s

#16 [builder 5/6] COPY . .
#16 sha256:f3077cc6506c589bf5b1c7ec4fc76f73260a89dad08354269b10baa4a9a98eae
#16 DONE 0.2s

#17 [builder 6/6] RUN npm run build
#17 sha256:c2ffc86e8cf1fc635561cad5f81e7a867ae588b3072d0a560fc0cfa441e13c46
#17 0.488 
#17 0.488 > gitlab-workflow@0.1.0 build /app
#17 0.488 > react-scripts build
#17 0.488 
#17 2.147 Creating an optimized production build...
#17 11.53 Compiled successfully.
#17 11.53 
#17 11.53 File sizes after gzip:
#17 11.53 
#17 11.54   41.34 KB  build/static/js/2.a1c765de.chunk.js
#17 11.54   1.57 KB   build/static/js/3.5f1cecaf.chunk.js
#17 11.54   1.17 KB   build/static/js/runtime-main.d5610362.js
#17 11.54   605 B     build/static/js/main.fedc4cc0.chunk.js
#17 11.54   574 B     build/static/css/main.9d5b29c0.chunk.css
#17 11.54 
#17 11.55 The project was built assuming it is hosted at /.
#17 11.55 You can control this with the homepage field in your package.json.
#17 11.55 
#17 11.55 The build folder is ready to be deployed.
#17 11.55 You may serve it with a static server:
#17 11.55 
#17 11.55   npm install -g serve
#17 11.55   serve -s build
#17 11.55 
#17 11.55 Find out more about deployment here:
#17 11.55 
#17 11.55   https://cra.link/deployment
#17 11.55 
#17 DONE 12.0s

#8 [stage-1 3/7] COPY ./nginx/envsubst-on-index.html.sh /docker-entrypoint.d/00-envsubst-on-index.html.sh
#8 sha256:aed08b04fc92fc3c4c49c128541059501f8558a3cb572bfb137d37e63a0534fc
#8 CACHED

#9 [stage-1 4/7] RUN chmod +x /docker-entrypoint.d/00-envsubst-on-index.html.sh
#9 sha256:07b379b1915337022df29b5228756dd3f90be375fc0806c363208f9a38a628ab
#9 CACHED

#10 [stage-1 5/7] WORKDIR /usr/share/nginx/html
#10 sha256:78af3ad2191e1e5b177aa2d602aebdc4dd49e07bd6b3eb0135c546febc54307c
#10 CACHED

#11 [stage-1 6/7] RUN rm -rf ./*
#11 sha256:43c10500269fdf8d643bb7d1ceff3fd93083ed1c1228475757d23c647766ee05
#11 CACHED

#7 [stage-1 2/7] COPY ./nginx/nginx.conf.template /etc/nginx/templates/default.conf.template
#7 sha256:54f375034c1bd8409bd87aec2d2677e8987c9813d309f665a8a311c3dc595487
#7 CACHED

#18 [stage-1 7/7] COPY --from=builder /app/build .
#18 sha256:ea76ce64d7316302cab2a05e21fce8db0f41b676c56539183b331a2ac1ca0ca1
#18 CACHED

#20 exporting to image
#20 sha256:e8c613e07b0b7ff33893b694f7759a10d42e180f2b4dc349fb57dc6b71dcab00
#20 exporting layers done
#20 writing image sha256:1a281131ae4ef806ba919cbb46ec227868d8edf5752cc5d703ed7c1107d2f9b5 done
#20 naming to gitlab.example.com:5050/petermarcoen/gitlab-workflow:latest done
#20 DONE 0.0s

#21 exporting cache
#21 sha256:2700d4ef94dee473593c5c614b55b2dedcca7893909811a8f2b48291a1f581e4
#21 preparing build cache for export done
#21 DONE 0.0s
$ docker push $CI_REGISTRY_IMAGE:latest
The push refers to repository [gitlab.example.com:5050/petermarcoen/gitlab-workflow]
d035df37e9ab: Preparing
ef26d49e08fe: Preparing
5f70bf18a086: Preparing
e7da80ad5b1e: Preparing
d29b2ccab66f: Preparing
d58fba3b3975: Preparing
4689e8eca613: Preparing
3480549413ea: Preparing
3c369314e003: Preparing
4531e200ac8d: Preparing
ed3fe3f2b59f: Preparing
b2d5eeeaba3a: Preparing
4689e8eca613: Waiting
3480549413ea: Waiting
3c369314e003: Waiting
4531e200ac8d: Waiting
ed3fe3f2b59f: Waiting
b2d5eeeaba3a: Waiting
d58fba3b3975: Waiting
ef26d49e08fe: Layer already exists
d29b2ccab66f: Layer already exists
5f70bf18a086: Layer already exists
e7da80ad5b1e: Layer already exists
d035df37e9ab: Layer already exists
4689e8eca613: Layer already exists
d58fba3b3975: Layer already exists
ed3fe3f2b59f: Layer already exists
3480549413ea: Layer already exists
3c369314e003: Layer already exists
4531e200ac8d: Layer already exists
b2d5eeeaba3a: Layer already exists
latest: digest: sha256:a52824310d5a47c521f9b31dab5474fba93ee6216ac3a646736c1191132a50c4 size: 2813
section_end:1619689494:step_script
section_start:1619689494:cleanup_file_variables
Cleaning up file based variables
section_end:1619689495:cleanup_file_variables
Job succeeded

Notice that for example step [builder 4/6] is not CACHED.

I have the same problem after using Docker in Docker and gitlab-runner as container.

Hello, I am facing the same issue and this confuses me. Were you able to resolve this issue @pmarcoen @mrostami

Thank you very much.

No, I just accepted it for now, still hoping for an answer from someone in the community.

Post all of the configuration files you created here.

Same for me.
I started digging into the cache-from option and with or without DOCKER_BUILDKIT + BUILDKIT_INLINE_CACHE=1 args but doesn’t affect anything.

Not sure if it’s your issue, but I’ve found the that docker image layer cache on GitLab is very particular. If there’s any file that changes in the layer, this will invalidate the cache, and it’s often something really silly. I see you’re using COPY . .. This is often the cause of cache invalidation because it copies things it doesn’t need to, and then the builder sees this as a file system change. Try dialing in your COPY, or deploy a dockerignore file that filters out anything not essential to your build. I’ve found things like log files, file timestamps, file permissions, and file ownership to all invalidate my layer caches. Mind you I’m not using build kit currently.

Additionally, in my work Gitlab we use a modified version of Gitlab’s autobuild image to do our building. One thing that we do is specify several images docker can pull caches from, and we manually pull these before running the build. This is truncated, but it’s something like this:

  image_duplicate="${CI_APPLICATION_REPOSITORY}${repo_path}:$CI_COMMIT_SHA"
  image_previous="${CI_APPLICATION_REPOSITORY}${repo_path}:$CI_COMMIT_BEFORE_SHA"
  image_latest="${CI_APPLICATION_REPOSITORY}${repo_path}:$(docker_tag_format "$CI_COMMIT_REF_NAME")"
 
....

 docker image pull --quiet "$image_duplicate" || \
    docker image pull --quiet "$image_previous" || \
    docker image pull --quiet "$image_latest" || \
    echo "No previously cached image found. The docker build will proceed without using a cached image"

....

  args+=(
    --tag="$image_duplicate"
    --tag="$image_latest"
  )

  cache_from_args+=(
    --cache-from="$image_duplicate"
    --cache-from="$image_previous"
    --cache-from="$image_latest"
  )

docker build "${cache_from_args[@]}" "${args[@]}" "${build_args[@]}" .

docker push "$image_duplicate"
docker push "$image_latest"

This has worked pretty well!

Another thing we optionally allow is building each layer of a multi-layer image separately and pushing it. This slows down the initial build slightly, but subsequent builds tend to run quicker. This is something that’s setup as optional and is only useful if your lower layers don’t change often, which is often not the case. An example where this is useful is where you have a builder step that requires a lot of boiler plate custom setup that is specific to this project (so just making it a custom build image doesn’t make a lot of sense). You can structure your image something like:

FROM baseimage AS build_base
RUN install basic setup stuff

FROM builder_base AS builder
COPY code_src ./app
RUN make build

FROM alpine:3.14
COPY --from=builder /app/main .

You would then do two docker build. The first will use --target builder_base, and will push this as a separate image. Since it doesn’t change often, if at all, it will cache a lot and the build step will only be copying code and building. I also use a lot of custom images. We write Go apis, so I have custom golang:alpine images that have a lot of the basic modules pre-installed (such as Ginkgo, Labstack Echo, gorilla, etc). This cuts down image build time a lot as these are large libraries that don’t change too often. If a developer really wants a newer version than what’s provided, they can change the version in their go.mod file and bloat their build time, but even then it tends to be quicker as a lot of the dependencies are the same.

Hope some of these ideas help

I have the same problem after using Docker. I tried to Flush DNS Cache and then restart the server. Its done.

I’ve been stuck on this for a while, but finally got it working. There were a few things I needed to do.

• Installed dive. Extremely useful for diving into a docker image to see the contents and digest of all the layers in a docker image.
• Compared a locally built docker image with one built in Gitlab CI to inspect the differences.
• Updated .dockerignore to exclude any files which weren’t available on a fresh checkout (as well as others which should have been excluded anyway. ref)
• Noticed that the file permissions in the image built on Gitlab were different that the one built locally. Did some digging and found this issue.
• Applied the suggested change in this comment to ensure the proper permissions in the Gitlab build on a shared runner

My final gitlab job looks like this

docker-build:
  stage: build
  image: docker:20.10.14-git
  needs: []
  services:
    - docker:20.10.14-dind
  variables:
    DOCKER_BUILDKIT: '1'
    DOCKER_HOST: tcp://docker:2375
    DOCKER_TLS_CERTDIR: ''
  before_script:
    - git checkout-index --all --prefix umask-fix-checkout/
    - cd umask-fix-checkout
    - docker login registry.gitlab.com -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD
  script:
    - docker build
      --target builder
      --cache-from $CI_REGISTRY_IMAGE:builder
      --tag $CI_REGISTRY_IMAGE:builder
      --build-arg BUILDKIT_INLINE_CACHE=1
      "."
    - docker build
      --target dev
      --cache-from $CI_REGISTRY_IMAGE:dev
      --tag $CI_REGISTRY_IMAGE:dev
      --build-arg BUILDKIT_INLINE_CACHE=1
      "."
    - docker push $CI_REGISTRY_IMAGE:builder
    - docker push $CI_REGISTRY_IMAGE:dev

I had same issue and I solved it by adding .git* in .dockerignore file. I realized .git directory differs from every run. That’s why if you have something like COPY . in your dockerfile that causes cache invalidated. Hope this helps.

In my case in was enough to add:

variables:
  DOCKER_BUILDKIT: '1'

  script:
    - docker build ........  --build-arg BUILDKIT_INLINE_CACHE=1