Docker build takes long time on exporting layers

I am trying to speed up the image build process in my pipeline that uses docker-dind. In the pipeline, the time taken for exporting layers during the image build reaches 276.4 seconds.

#15 exporting to image
#15 exporting layers
#15 exporting layers 276.4s done
#15 writing image sha256:c70b1fbdf84fe2323fbe85d6e546f162bd3dcb5c195c8ddefbb1af87eb1e2084 done
#15 naming to gitlab.example.com:v1.0.1.1-release 0.0s done
#15 DONE 276.4s
WARNING: current commit information was not captured by the build: git was not found in the system: exec: "git": executable file not found in $PATH
$ docker push $IMAGE_FULL

This is very different from when I try to run the same command in the shell on my GitLab Runner VM, which only takes a few seconds due to caching.

What I have tried in my pipeline job is using BuildKit and --cache-from as follows:

build-image:
  image: 
    name: docker:24.0.7
  stage: build-image
  services:
    - docker:24.0.7-dind
  variables:
    DOCKER_HOST: "tcp://localhost:2375"
    DOCKER_BUILDKIT: '1'
  before_script:
    - update-ca-certificates && apk add git
    - export IMAGE_FULL=$HARBOR_REGISTRY/$IMAGE_NAME:$STAGING_VERSION
    - git checkout-index --all --prefix umask-fix-checkout/
    - cd umask-fix-checkout
    - if [[ $CI_COMMIT_TAG != '' ]]; then export IMAGE_FULL=$HARBOR_REGISTRY/$IMAGE_NAME:$CI_COMMIT_TAG; fi;
    - if [[ $CI_COMMIT_TAG != '' ]]; then export ENV_APPLICATION=".env-production"; fi;
  script:
    - docker login -u $HARBOR_REGISTRY_USERNAME -p $HARBOR_REGISTRY_PASSWORD $HARBOR_REGISTRY
    - echo "Building Image $IMAGE_FULL"
    - cp $ENV_APPLICATION .env
    # - docker build -t $IMAGE_FULL .
    - time docker build 
        --cache-from $STAGING_VERSION:build 
        -t $IMAGE_FULL
        --build-arg BUILDKIT_INLINE_CACHE=1
        "."
    - docker push $IMAGE_FULL

Here is my Dockerfile:

# Stage 1: Build the Next.js app
FROM node:20.12-alpine AS build
WORKDIR /app
COPY package.json package-lock.json ./
RUN npm ci --prefer-offline --no-audit
COPY . .
RUN npm run build

# Stage 2: Create a lightweight image to run the Next.js app
FROM node:20.12-alpine
WORKDIR /app
COPY --from=build /app/package.json /app/package-lock.json ./
COPY --from=build /app/.next ./.next
COPY --from=build /app/public ./public

RUN npm ci --prefer-offline --no-audit --omit=dev

EXPOSE 3000

CMD ["npm", "start"]

However, the process of exporting layers always takes a long time, and pushing to the registry also takes a considerable amount of time, which is different from when I try it through the shell on the GitLab Runner VM.

As a note, I have also tried changing the runner executor to shell, but the time required to build the image did not change / did not get any faster.