Docker build not using cache

I am trying to speed up my Docker build by using the --cache-from option.
This works on my local machine but not on Gitlab (EE) and I ran out of ideas. Can anyone please take a look and maybe see what I am missing?

This is my Dockerfile, notice that it consists of multiple stages:

# Build application
FROM node:12 AS builder
WORKDIR /app
COPY package*.json ./
RUN npm install
COPY . .
RUN npm run build

# Deploy on NGINX
FROM nginx:alpine
COPY ./nginx/nginx.conf.template /etc/nginx/templates/default.conf.template
COPY ./nginx/envsubst-on-index.html.sh /docker-entrypoint.d/00-envsubst-on-index.html.sh
RUN chmod +x /docker-entrypoint.d/00-envsubst-on-index.html.sh
WORKDIR /usr/share/nginx/html
RUN rm -rf ./*
COPY --from=builder /app/build .

Now if I run the commands below locally using the latest image stored in the Gitlab registry the build is incredibly fast since everything is cached.
Notice that I have purposely removed all local images and logged in to my Gitlab registry.

PS D:\temp\gitlab-workflow> docker image ls -a
REPOSITORY   TAG       IMAGE ID   CREATED   SIZE
PS D:\temp\gitlab-workflow> docker login gitlab.example.com:5050
Authenticating with existing credentials...
Login Succeeded
PS D:\temp\gitlab-workflow> docker --version
Docker version 20.10.5, build 55c4c88
PS D:\temp\gitlab-workflow>
PS D:\temp\gitlab-workflow>
PS D:\temp\gitlab-workflow> docker build --build-arg BUILDKIT_INLINE_CACHE=1 --cache-from gitlab.example.com:5050/petermarcoen/gitlab-workflow:latest -t gitlab.example.com:5050/petermarcoen/gitlab-workflow:latest .
[+] Building 27.7s (24/24) FINISHED
 => [internal] load build definition from Dockerfile                                                                                                                                                 0.0s 
 => => transferring dockerfile: 518B                                                                                                                                                                 0.0s 
 => [internal] load .dockerignore                                                                                                                                                                    0.0s 
 => => transferring context: 2B                                                                                                                                                                      0.0s 
 => [internal] load metadata for docker.io/library/nginx:alpine                                                                                                                                      1.7s 
 => [internal] load metadata for docker.io/library/node:12                                                                                                                                           1.7s 
 => [auth] library/nginx:pull token for registry-1.docker.io                                                                                                                                         0.0s 
 => [auth] library/node:pull token for registry-1.docker.io                                                                                                                                          0.0s 
 => importing cache manifest from gitlab.example.com:5050/petermarcoen/gitlab-workflow:latest                                                                                                          1.1s 
 => [stage-1 1/7] FROM docker.io/library/nginx:alpine@sha256:07ab71a2c8e4ecb19a5a5abcfb3a4f175946c001c8af288b1aa766d67b0d05d2                                                                        0.0s 
 => [builder 1/6] FROM docker.io/library/node:12@sha256:609103746810535f5a3a987a26ba4ce95d96225d28e9d6228faa5aa331980f37                                                                             0.0s 
 => [internal] load build context                                                                                                                                                                    4.2s 
 => [auth] petermarcoen/gitlab-workflow:pull token for gitlab.example.com:5050                                                                                                                         0.0s 
 => CACHED [builder 2/6] WORKDIR /app                                                                                                                                                                0.0s 
 => CACHED [builder 3/6] COPY package*.json ./                                                                                                                                                       0.0s 
 => CACHED [builder 4/6] RUN npm install                                                                                                                                                             0.0s 
 => [builder 5/6] COPY . .                                                                                                                                                                           6.8s 
 => [builder 6/6] RUN npm run build                                                                                                                                                                 12.8s 
 => CACHED [stage-1 2/7] COPY ./nginx/nginx.conf.template /etc/nginx/templates/default.conf.template                                                                                                 0.0s 
 => CACHED [stage-1 3/7] COPY ./nginx/envsubst-on-index.html.sh /docker-entrypoint.d/00-envsubst-on-index.html.sh                                                                                    0.0s 
 => CACHED [stage-1 4/7] RUN chmod +x /docker-entrypoint.d/00-envsubst-on-index.html.sh                                                                                                              0.0s 
 => CACHED [stage-1 5/7] WORKDIR /usr/share/nginx/html                                                                                                                                               0.0s 
 => CACHED [stage-1 6/7] RUN rm -rf ./*                                                                                                                                                              0.0s 
 => [stage-1 7/7] COPY --from=builder /app/build .                                                                                                                                                   0.1s 
 => exporting to image                                                                                                                                                                               0.0s 
 => => exporting layers                                                                                                                                                                              0.0s 
 => => writing image sha256:48e4c21e9ad8377376041055fc6c184f64c0c531c0171c4d4a5a027497f9c0bf                                                                                                         0.0s 
 => => naming to gitlab.example.com:5050/petermarcoen/gitlab-workflow:latest                                                                                                                           0.0s 
 => exporting cache                                                                                                                                                                                  0.0s 
 => => preparing build cache for export                                                                                                                                                              0.0s 

Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them
PS D:\temp\gitlab-workflow>

Notice that for example step [builder 4/6] is CACHED.
In my gitlab-ci.yml I (think I) do the same thing:

variables:
  DOCKER_BUILDKIT: 1
stages:
  - build
build:
  image: docker:20.10.5
  stage: build
  tags:
    - docker
  services:
    - name: docker:stable-dind
  script:
    - echo "$CI_REGISTRY_IMAGE:latest"
    - docker --version
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
    - docker pull $CI_REGISTRY_IMAGE:latest || true # Pull latest image to leverage layer caching, true to avoid failing if image is not found
    - docker build --build-arg BUILDKIT_INLINE_CACHE=1 --cache-from $CI_REGISTRY_IMAGE:latest -t $CI_REGISTRY_IMAGE:latest . # Create an image tagged latest
    - docker push $CI_REGISTRY_IMAGE:latest # Push the image to the Gitlab registry

But here the cache is not used and everything is done from scratch:

Running with gitlab-runner 13.10.0 (54944146)
  on master-3 YT2nPraF
section_start:1619689332:prepare_executor
Preparing the "docker" executor
Using Docker executor with image docker:20.10.5 ...
Starting service docker:stable-dind ...
Pulling docker image docker:stable-dind ...
Using docker image sha256:6f19136cf89d49aee8fa560eaab20e034c6ee17bbce55a9d33e1fbe5baa35b03 for docker:stable-dind with digest docker@sha256:4972457c6a8a4309f9796fc3c8fd288923045ba0e214c102d92b70be99e249d1 ...
Waiting for services to be up and running...
Pulling docker image docker:20.10.5 ...
Using docker image sha256:1588477122de4fdfe9fcb9ddeeee6ac6b93e9e05a65c68a6e22add0a98b8e0fe for docker:20.10.5 with digest docker@sha256:7ed427295687586039ff3433bb9b4419c5cf1e6294025dadf7641126665a78f5 ...
section_end:1619689348:prepare_executor
section_start:1619689348:prepare_script
Preparing environment
Running on runner-yt2npraf-project-4-concurrent-0 via c2c25f0e33b6...
section_end:1619689348:prepare_script
section_start:1619689348:get_sources
Getting source from Git repository
Fetching changes with git depth set to 50...
Reinitialized existing Git repository in /builds/petermarcoen/gitlab-workflow/.git/
Checking out b6f85f62 as master...

Skipping Git submodules setup
section_end:1619689350:get_sources
section_start:1619689350:step_script
Executing "step_script" stage of the job script
Using docker image sha256:1588477122de4fdfe9fcb9ddeeee6ac6b93e9e05a65c68a6e22add0a98b8e0fe for docker:20.10.5 with digest docker@sha256:7ed427295687586039ff3433bb9b4419c5cf1e6294025dadf7641126665a78f5 ...
$ echo "$CI_REGISTRY_IMAGE:latest"
gitlab.example.com:5050/petermarcoen/gitlab-workflow:latest
$ docker --version
Docker version 20.10.5, build 55c4c88
$ docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded
$ docker pull $CI_REGISTRY_IMAGE:latest || true
latest: Pulling from petermarcoen/gitlab-workflow
540db60ca938: Pulling fs layer
...
f8277735e4e6: Pull complete
Digest: sha256:9866109bf8d5e0bb7464c296d26894656fc5ca05010363968db2b7b20e21ab91
Status: Downloaded newer image for gitlab.example.com:5050/petermarcoen/gitlab-workflow:latest
gitlab.example.com:5050/petermarcoen/gitlab-workflow:latest
$ docker build --build-arg BUILDKIT_INLINE_CACHE=1 --cache-from $CI_REGISTRY_IMAGE:latest -t $CI_REGISTRY_IMAGE:latest .
#1 [internal] load .dockerignore
#1 sha256:e60682f2aa9fc4906b9bada9c7560194e970b39baa2b520526f4bd88285e81b4
#1 transferring context: 2B done
#1 DONE 0.1s

#2 [internal] load build definition from Dockerfile
#2 sha256:7c337464965d70e2254ba0a93736318dd78dd915600cf36665260af68bf62844
#2 transferring dockerfile: 503B done
#2 DONE 0.1s

#4 [internal] load metadata for docker.io/library/node:12
#4 sha256:55906eb64e7bc4d69cef5ad3f3679feeb2af997de97f0dab4394b48f2c72efd2
#4 DONE 1.6s

#3 [internal] load metadata for docker.io/library/nginx:alpine
#3 sha256:b001d263a254f0e4960d52c837d5764774ef80ad3878c61304052afb6e0e9af2
#3 DONE 1.8s

#19 importing cache manifest from gitlab.example.com:5050/petermarcoen/gitlab-workflow:latest
#19 sha256:cb3fd8bb86fa9e8874fecf3ea295257e0ebcb4b1eb5cf3d6d3e7a515569cb16c
#19 DONE 0.0s

#5 [stage-1 1/7] FROM docker.io/library/nginx:alpine@sha256:07ab71a2c8e4ecb19a5a5abcfb3a4f175946c001c8af288b1aa766d67b0d05d2
#5 sha256:9e7cf3b35eef66df544c5643e6f1964f9ed0e9beae5c6a12bd7763d532039238
#5 DONE 0.0s

#6 [internal] load build context
#6 sha256:2a6388e58f30fc93807b1f1a4797c36879dc0cff5ac162a11e608a6a8c6b5804
#6 transferring context: 2.11MB 0.1s done
#6 DONE 0.1s

#12 [builder 1/6] FROM docker.io/library/node:12@sha256:609103746810535f5a3a987a26ba4ce95d96225d28e9d6228faa5aa331980f37
#12 sha256:049f879149ced3f210db72f418337b07e9e993775dca36fd9a970b248c8fec14
#12 resolve docker.io/library/node:12@sha256:609103746810535f5a3a987a26ba4ce95d96225d28e9d6228faa5aa331980f37 done
#12 sha256:3ff9a21f91c215efa3b65ae958df9bd5011e55301849ba0789e2788575b1c0df 2.21kB / 2.21kB done
...
#12 extracting sha256:20f8d3e06d470f5659ac84232f4d5b309ad360a0b87a01f8042a020296616d69 done
#12 DONE 44.0s

#13 [builder 2/6] WORKDIR /app
#13 sha256:7bd1b19391b06eddb9e1b12d94cf7c192f6ef4798bc1e7315b4117bba5103b72
#13 DONE 24.6s

#14 [builder 3/6] COPY package*.json ./
#14 sha256:60a7b5046cab58d2e81a646151e00b19dc1074bccfef323f727cd15fb816caa8
#14 DONE 0.1s

#15 [builder 4/6] RUN npm install
#15 sha256:ffda396506740db21ca2da5b5ac875fc2cee6c670777216d32490345c62cb8fa
#15 49.58 
#15 49.58 > core-js@2.6.12 postinstall /app/node_modules/babel-runtime/node_modules/core-js
#15 49.58 > node -e "try{require('./postinstall')}catch(e){}"
#15 49.58 
#15 49.71 Thank you for using core-js ( https://github.com/zloirock/core-js ) for polyfilling JavaScript standard library!
#15 49.71 
#15 49.71 The project needs your help! Please consider supporting of core-js on Open Collective or Patreon: 
#15 49.71 > https://opencollective.com/core-js 
#15 49.71 > https://www.patreon.com/zloirock 
#15 49.71 
#15 49.71 Also, the author of core-js ( https://github.com/zloirock ) is looking for a good job -)
#15 49.71 
#15 49.84 
#15 49.84 > core-js@3.10.2 postinstall /app/node_modules/core-js
#15 49.84 > node -e "try{require('./postinstall')}catch(e){}"
#15 49.84 
#15 49.91 Thank you for using core-js ( https://github.com/zloirock/core-js ) for polyfilling JavaScript standard library!
#15 49.91 
#15 49.91 The project needs your help! Please consider supporting of core-js on Open Collective or Patreon: 
#15 49.91 > https://opencollective.com/core-js 
#15 49.91 > https://www.patreon.com/zloirock 
#15 49.91 
#15 49.91 Also, the author of core-js ( https://github.com/zloirock ) is looking for a good job -)
#15 49.91 
#15 49.96 
#15 49.96 > core-js-pure@3.10.2 postinstall /app/node_modules/core-js-pure
#15 49.96 > node -e "try{require('./postinstall')}catch(e){}"
#15 49.96 
#15 50.15 Thank you for using core-js ( https://github.com/zloirock/core-js ) for polyfilling JavaScript standard library!
#15 50.15 
#15 50.15 The project needs your help! Please consider supporting of core-js on Open Collective or Patreon: 
#15 50.15 > https://opencollective.com/core-js 
#15 50.15 > https://www.patreon.com/zloirock 
#15 50.15 
#15 50.15 Also, the author of core-js ( https://github.com/zloirock ) is looking for a good job -)
#15 50.15 
#15 50.22 
#15 50.22 > ejs@2.7.4 postinstall /app/node_modules/ejs
#15 50.22 > node ./postinstall.js
#15 50.22 
#15 50.33 Thank you for installing e[35mEJS: built with the e[32mJake JavaScript build tool (e[32mhttps://jakejs.com/)
#15 50.33 
#15 53.32 npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@1.2.13 (node_modules/webpack-dev-server/node_modules/fsevents):
#15 53.32 npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@1.2.13: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})
#15 53.34 npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@1.2.13 (node_modules/watchpack-chokidar2/node_modules/fsevents):
#15 53.34 npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@1.2.13: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})
#15 53.35 npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@2.3.2 (node_modules/fsevents):
#15 53.35 npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@2.3.2: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})
#15 53.35 
#15 53.36 added 1949 packages from 765 contributors and audited 1952 packages in 52.535s
#15 55.11 
#15 55.11 135 packages are looking for funding
#15 55.11   run `npm fund` for details
#15 55.11 
#15 55.11 found 0 vulnerabilities
#15 55.11 
#15 DONE 56.7s

#16 [builder 5/6] COPY . .
#16 sha256:f3077cc6506c589bf5b1c7ec4fc76f73260a89dad08354269b10baa4a9a98eae
#16 DONE 0.2s

#17 [builder 6/6] RUN npm run build
#17 sha256:c2ffc86e8cf1fc635561cad5f81e7a867ae588b3072d0a560fc0cfa441e13c46
#17 0.488 
#17 0.488 > gitlab-workflow@0.1.0 build /app
#17 0.488 > react-scripts build
#17 0.488 
#17 2.147 Creating an optimized production build...
#17 11.53 Compiled successfully.
#17 11.53 
#17 11.53 File sizes after gzip:
#17 11.53 
#17 11.54   41.34 KB  build/static/js/2.a1c765de.chunk.js
#17 11.54   1.57 KB   build/static/js/3.5f1cecaf.chunk.js
#17 11.54   1.17 KB   build/static/js/runtime-main.d5610362.js
#17 11.54   605 B     build/static/js/main.fedc4cc0.chunk.js
#17 11.54   574 B     build/static/css/main.9d5b29c0.chunk.css
#17 11.54 
#17 11.55 The project was built assuming it is hosted at /.
#17 11.55 You can control this with the homepage field in your package.json.
#17 11.55 
#17 11.55 The build folder is ready to be deployed.
#17 11.55 You may serve it with a static server:
#17 11.55 
#17 11.55   npm install -g serve
#17 11.55   serve -s build
#17 11.55 
#17 11.55 Find out more about deployment here:
#17 11.55 
#17 11.55   https://cra.link/deployment
#17 11.55 
#17 DONE 12.0s

#8 [stage-1 3/7] COPY ./nginx/envsubst-on-index.html.sh /docker-entrypoint.d/00-envsubst-on-index.html.sh
#8 sha256:aed08b04fc92fc3c4c49c128541059501f8558a3cb572bfb137d37e63a0534fc
#8 CACHED

#9 [stage-1 4/7] RUN chmod +x /docker-entrypoint.d/00-envsubst-on-index.html.sh
#9 sha256:07b379b1915337022df29b5228756dd3f90be375fc0806c363208f9a38a628ab
#9 CACHED

#10 [stage-1 5/7] WORKDIR /usr/share/nginx/html
#10 sha256:78af3ad2191e1e5b177aa2d602aebdc4dd49e07bd6b3eb0135c546febc54307c
#10 CACHED

#11 [stage-1 6/7] RUN rm -rf ./*
#11 sha256:43c10500269fdf8d643bb7d1ceff3fd93083ed1c1228475757d23c647766ee05
#11 CACHED

#7 [stage-1 2/7] COPY ./nginx/nginx.conf.template /etc/nginx/templates/default.conf.template
#7 sha256:54f375034c1bd8409bd87aec2d2677e8987c9813d309f665a8a311c3dc595487
#7 CACHED

#18 [stage-1 7/7] COPY --from=builder /app/build .
#18 sha256:ea76ce64d7316302cab2a05e21fce8db0f41b676c56539183b331a2ac1ca0ca1
#18 CACHED

#20 exporting to image
#20 sha256:e8c613e07b0b7ff33893b694f7759a10d42e180f2b4dc349fb57dc6b71dcab00
#20 exporting layers done
#20 writing image sha256:1a281131ae4ef806ba919cbb46ec227868d8edf5752cc5d703ed7c1107d2f9b5 done
#20 naming to gitlab.example.com:5050/petermarcoen/gitlab-workflow:latest done
#20 DONE 0.0s

#21 exporting cache
#21 sha256:2700d4ef94dee473593c5c614b55b2dedcca7893909811a8f2b48291a1f581e4
#21 preparing build cache for export done
#21 DONE 0.0s
$ docker push $CI_REGISTRY_IMAGE:latest
The push refers to repository [gitlab.example.com:5050/petermarcoen/gitlab-workflow]
d035df37e9ab: Preparing
ef26d49e08fe: Preparing
5f70bf18a086: Preparing
e7da80ad5b1e: Preparing
d29b2ccab66f: Preparing
d58fba3b3975: Preparing
4689e8eca613: Preparing
3480549413ea: Preparing
3c369314e003: Preparing
4531e200ac8d: Preparing
ed3fe3f2b59f: Preparing
b2d5eeeaba3a: Preparing
4689e8eca613: Waiting
3480549413ea: Waiting
3c369314e003: Waiting
4531e200ac8d: Waiting
ed3fe3f2b59f: Waiting
b2d5eeeaba3a: Waiting
d58fba3b3975: Waiting
ef26d49e08fe: Layer already exists
d29b2ccab66f: Layer already exists
5f70bf18a086: Layer already exists
e7da80ad5b1e: Layer already exists
d035df37e9ab: Layer already exists
4689e8eca613: Layer already exists
d58fba3b3975: Layer already exists
ed3fe3f2b59f: Layer already exists
3480549413ea: Layer already exists
3c369314e003: Layer already exists
4531e200ac8d: Layer already exists
b2d5eeeaba3a: Layer already exists
latest: digest: sha256:a52824310d5a47c521f9b31dab5474fba93ee6216ac3a646736c1191132a50c4 size: 2813
section_end:1619689494:step_script
section_start:1619689494:cleanup_file_variables
Cleaning up file based variables
section_end:1619689495:cleanup_file_variables
Job succeeded

Notice that for example step [builder 4/6] is not CACHED.