Multi-platform builds not working no matter what is tried

:hugs: Please help fill in this template with all the details to help others help you more efficiently. Use formatting blocks for code, config, logs and ensure to remove sensitive data.

Problem to solve

I’m trying to build a multi-platform image (eg linux/amd64, linux/arm64/v8). I could post my gitlab-cicd.yml file but that will not be helpful. I’ve tried every single combination of suggestions online from the last 5 years. This includes setting driver to docker-container, building custom contexts, using --push and/or --pull in the docker buildx build command, and upgrading to docker:25.0.3 and others.

No matter what I try I cant get the image (which apparently builds fine) to either show in the local registry or push to the GitLab container registry. EVERY single piece of help online I’ve found where someone says ‘All I had to do was…’ has not worked for me. Can someone post a hello world .gitlab-ci.yml file that builds multi-arch images and posts them to a registry or gets them to show in a docker images command. I’m truly at a loss as to what is going on

BTW, using latest Docker Desktop on my Mac M3 … I can get multi-arch images to build and appear in a ‘docker images’ command. What is going on?

Describe your question in as much detail as possible:

  • What are you seeing, and how does that differ from what you expect to see?
  • Consider including screenshots, error messages, and/or other helpful visuals

Steps to reproduce

I tried to paste my .gitlab-ci.yml file here… but got a 422 error. I tried to upload/attach the file and still no luck. If you email swsides@txcorp.com I can send it directly. Sorry about that…

Add the CI/CD configuration from .gitlab-ci.yml and other configuration if relevant (e.g. docker-compose.yml). Alternatively, create a public GitLab.com example project that provides all necessary files to reproduce the question.

Versions

Please select whether options apply, and add the version information.

  • Self-managed
  • GitLab.com SaaS
  • Self-hosted Runners

Versions

Helpful resources

  1. Before opening a new topic, make sure to search for keywords in the forum search
  2. Check the GitLab Runner and GitLab projects for existing issues. If you encounter a bug, please create a bug report issue.
  3. Troubleshooting docs: GitLab Runner, self-managed GitLab instances.

Thanks for taking the time to be thorough in your request, it really helps! :blush:Preformatted text

image: docker:latest

services:

  • docker:dind

variables:
DOCKER_HOST: tcp://docker:2375/
DOCKER_DRIVER: overlay2

before_script:

  • docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY

build:
stage: build
script:
- docker run --rm --privileged multiarch/qemu-user-static --reset -p yes
- docker buildx create --name mybuilder --use
- docker buildx build --platform linux/amd64,linux/arm64/v8 --tag $CI_REGISTRY_IMAGE:latest --push .
I can suggest a basic setup for your .gitlab-ci.yml file that should help with building and pushing images that work on different types of computers (like Intel and ARM processors). This setup assumes that you already have Docker and a GitLab Runner ready for handling builds for different platforms.
Here’s a simplified explanation of what to do:
Use the latest version of Docker along with Docker-in-Docker to make sure you can use the newest features.
Before you start building your Docker images, sign in to the GitLab Container Registry using the login details GitLab provides automatically.
Use a special container called multiarch/qemu-user-static to allow you to build ARM images even if you’re not using an ARM computer.
Set up a new Docker buildx builder named mybuilder and start using it.
Build your Docker image so it can work on both amd64 (standard Intel and AMD processors) and arm64/v8 (ARM processors) and then push this image to the GitLab Container Registry.
Make sure your GitLab Runner is set up to run Docker commands and has the right permissions. Also, check that your Docker Desktop app is up to date and set up correctly to work with GitLab CI/CD. This setup is a starting point, and you might need to adjust it based on the specific errors you encounter or the particular needs of your project.

Thank you so much for the reply. The qemup-user-static container sounds familar… I have downloaded it already, but perhaps I didnt use it correctly. I’ll try it again with your other suggestions. If I build correctly with your suggestions… will the image appear if I do ‘docker images’? And… will I then be able to do an explicit ‘docker push …’? Or… should the ‘docker buildx build…’ line include ‘–push’ ?

Also… I have the latest Docker Desktop and builds there (on my mac) definitely work without too much trouble. Why did you mention getting Docker Desktop for fixing my GitLab pipeline on a shared GitLab runner?

Here are the ‘important’ lines:

image: docker:25.0.3
docker run --rm --privileged multiarch/qemu-user-static --reset -p yes --install all
docker buildx create --platform linux/amd64,linux/arm64 --driver=docker-container --name=mybuilder --use --bootstrap

services:

  • name: docker:25.0.3-dind
    command: [“–experimental”]

docker buildx build --push --platform=linux/amd64,linux/arm64 -t “$CS_REPO/test2:$CS_IMAGES_VERSION” -f test.dkr --builder=mybuilder .

And like all my other attempts… the image apparently builds but it pushes an ‘empty’ image to the registry. This is draining me of the will to live. Help?

From what I gather, there could be a few things worth double-checking:
First, It’s crucial to make sure your Dockerfile and the whole setup are ready to handle builds for different types of processors, like Intel and ARM. Sometimes, this means you have to pick base images or dependencies that play nice with both kinds of tech.
You’ll want to ensure that when you tell Docker to build your image, the path you give it includes everything your application needs. If the path is wrong or missing stuff, you might end up with a build that’s not what you expected.
Also, debugging might help to add more detailed logs to your GitLab CI/CD pipeline. More info could give you clues on where things are getting tangled.
From my angle, you seem pretty close to cracking this. Sometimes, breaking down each step and looking at them one by one can help pinpoint the problem.

Thanks for the reply. So I’ve made a very simple test Docker image that essentially just downloads an alpine:3.14 image and sets me as the MAINTAINER. This builds multi-arch fine on my mac with latest Docker Desktop. The full log from the build on a shared linux gitlab runner. The docker buildx build … command I’m using for the GitLab pipeline has --push. So it directly pushes an image to the gitlab registry… but its simply empty. Not sure why this is so easy on Docker Desktop and so hard in GitLab pipeline. Does this log have any extra info that is helpful?

[0KRunning with gitlab-runner 16.9.1 (782c6ecb)e[0;m
e[0K  on blue-3.saas-linux-large-amd64.runners-manager.gitlab.com/default VxfMyqp3, system ID: s_a57dbf17d1f2e[0;m
e[0K  feature flags: FF_USE_IMPROVED_URL_MASKING:truee[0;m
section_start:1710184698:resolve_secrets
e[0Ke[0Ke[36;1mResolving secretse[0;me[0;m
section_end:1710184698:resolve_secrets
e[0Ksection_start:1710184698:prepare_executor
e[0Ke[0Ke[36;1mPreparing the "docker+machine" executore[0;me[0;m
e[0KUsing Docker executor with image docker:25.0.3 ...e[0;m
e[0KStarting service docker:25.0.3-dind ...e[0;m
e[0KPulling docker image docker:25.0.3-dind ...e[0;m
e[0KUsing docker image sha256:e5fbe8997fd9ff8f2894874c12a4fbc5fc1bb42d08a7db8433bbe09066562a2a for docker:25.0.3-dind with digest docker@sha256:0d70c541ee98e66b8f7ece8c0e9f7910732466e337a9087c2ac2868ef0775092 ...e[0;m
e[0KWaiting for services to be up and running (timeout 30 seconds)...e[0;m
e[0KPulling docker image docker:25.0.3 ...e[0;m
e[0KUsing docker image sha256:e5fbe8997fd9ff8f2894874c12a4fbc5fc1bb42d08a7db8433bbe09066562a2a for docker:25.0.3 with digest docker@sha256:0d70c541ee98e66b8f7ece8c0e9f7910732466e337a9087c2ac2868ef0775092 ...e[0;m
section_end:1710184727:prepare_executor
e[0Ksection_start:1710184727:prepare_script
e[0Ke[0Ke[36;1mPreparing environmente[0;me[0;m
Running on runner-vxfmyqp3-project-40608643-concurrent-0 via runner-vxfmyqp3-s-l-l-amd64-1710184654-d93e88b0...
section_end:1710184727:prepare_script
e[0Ksection_start:1710184727:get_sources
e[0Ke[0Ke[36;1mGetting source from Git repositorye[0;me[0;m
e[32;1mFetching changes with git depth set to 20...e[0;m
Initialized empty Git repository in /builds/txcorp/chemistream/chemistream-hpc-images/.git/
e[32;1mCreated fresh repository.e[0;m
e[32;1mChecking out f70d20a9 as detached HEAD (ref is refs/merge-requests/38/head)...e[0;m

e[32;1mSkipping Git submodules setupe[0;m
e[32;1m$ git remote set-url origin "${CI_REPOSITORY_URL}"e[0;m
section_end:1710184739:get_sources
e[0Ksection_start:1710184739:step_script
e[0Ke[0Ke[36;1mExecuting "step_script" stage of the job scripte[0;me[0;m
e[0KUsing docker image sha256:e5fbe8997fd9ff8f2894874c12a4fbc5fc1bb42d08a7db8433bbe09066562a2a for docker:25.0.3 with digest docker@sha256:0d70c541ee98e66b8f7ece8c0e9f7910732466e337a9087c2ac2868ef0775092 ...e[0;m
e[32;1m$ export CS_REPO="registry.gitlab.com/txcorp/chemistream/chemistream-hpc-images"e[0;m
e[32;1m$ echo "CS_REPO=$CS_REPO"e[0;m
CS_REPO=registry.gitlab.com/txcorp/chemistream/chemistream-hpc-images
e[32;1m$ echo "CS_IMAGES_VERSION=$CS_IMAGES_VERSION"e[0;m
CS_IMAGES_VERSION=latest
e[32;1m$ docker infoe[0;m
Client:
 Version:    25.0.3
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.12.1
    Path:     /usr/local/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.24.6
    Path:     /usr/local/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 0
 Server Version: 25.0.3
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: runc io.containerd.runc.v2
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 7c3aca7a610df76212171d200ca3811ff6096eb8
 runc version: v1.1.12-0-g51d5e94
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
 Kernel Version: 5.4.109+
 Operating System: Alpine Linux v3.19 (containerized)
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 31.37GiB
 Name: feaa4907b966
 ID: 9e3cdf19-640b-472c-8527-f1532498c71d
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: true
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
 Product License: Community Engine

WARNING: API is accessible on http://0.0.0.0:2375 without encryption.
         Access to the remote API is equivalent to root access on the host. Refer
         to the 'Docker daemon attack surface' section in the documentation for
         more information: https://docs.docker.com/go/attack-surface/
e[32;1m$ apk updatee[0;m
fetch https://dl-cdn.alpinelinux.org/alpine/v3.19/main/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.19/community/x86_64/APKINDEX.tar.gz
v3.19.1-217-g0e9c38378f6 [https://dl-cdn.alpinelinux.org/alpine/v3.19/main]
v3.19.1-219-g2e1eeb11dbd [https://dl-cdn.alpinelinux.org/alpine/v3.19/community]
OK: 22987 distinct packages available
e[32;1m$ apk add gite[0;m
OK: 42 MiB in 71 packages
e[32;1m$ echo ""e[0;m

e[32;1m$ echo ""e[0;m

e[32;1m$ echo "Buildx create for multi-platform builds"e[0;m
Buildx create for multi-platform builds
e[32;1m$ docker run --rm --privileged multiarch/qemu-user-static --reset -p yes --install alle[0;m
Unable to find image 'multiarch/qemu-user-static:latest' locally
latest: Pulling from multiarch/qemu-user-static
205dae5015e7: Pulling fs layer
816739e52091: Pulling fs layer
30abb83a18eb: Pulling fs layer
0657daef200b: Pulling fs layer
30c9c93f40b9: Pulling fs layer
0657daef200b: Waiting
30c9c93f40b9: Waiting
816739e52091: Verifying Checksum
816739e52091: Download complete
30abb83a18eb: Verifying Checksum
30abb83a18eb: Download complete
205dae5015e7: Verifying Checksum
205dae5015e7: Download complete
205dae5015e7: Pull complete
0657daef200b: Verifying Checksum
0657daef200b: Download complete
816739e52091: Pull complete
30abb83a18eb: Pull complete
0657daef200b: Pull complete
30c9c93f40b9: Verifying Checksum
30c9c93f40b9: Download complete
30c9c93f40b9: Pull complete
Digest: sha256:fe60359c92e86a43cc87b3d906006245f77bfc0565676b80004cc666e4feb9f0
Status: Downloaded newer image for multiarch/qemu-user-static:latest
getopt: unrecognized option '--install'
Setting /usr/bin/qemu-alpha-static as binfmt interpreter for alpha
Setting /usr/bin/qemu-arm-static as binfmt interpreter for arm
Setting /usr/bin/qemu-armeb-static as binfmt interpreter for armeb
Setting /usr/bin/qemu-sparc-static as binfmt interpreter for sparc
Setting /usr/bin/qemu-sparc32plus-static as binfmt interpreter for sparc32plus
Setting /usr/bin/qemu-sparc64-static as binfmt interpreter for sparc64
Setting /usr/bin/qemu-ppc-static as binfmt interpreter for ppc
Setting /usr/bin/qemu-ppc64-static as binfmt interpreter for ppc64
Setting /usr/bin/qemu-ppc64le-static as binfmt interpreter for ppc64le
Setting /usr/bin/qemu-m68k-static as binfmt interpreter for m68k
Setting /usr/bin/qemu-mips-static as binfmt interpreter for mips
Setting /usr/bin/qemu-mipsel-static as binfmt interpreter for mipsel
Setting /usr/bin/qemu-mipsn32-static as binfmt interpreter for mipsn32
Setting /usr/bin/qemu-mipsn32el-static as binfmt interpreter for mipsn32el
Setting /usr/bin/qemu-mips64-static as binfmt interpreter for mips64
Setting /usr/bin/qemu-mips64el-static as binfmt interpreter for mips64el
Setting /usr/bin/qemu-sh4-static as binfmt interpreter for sh4
Setting /usr/bin/qemu-sh4eb-static as binfmt interpreter for sh4eb
Setting /usr/bin/qemu-s390x-static as binfmt interpreter for s390x
Setting /usr/bin/qemu-aarch64-static as binfmt interpreter for aarch64
Setting /usr/bin/qemu-aarch64_be-static as binfmt interpreter for aarch64_be
Setting /usr/bin/qemu-hppa-static as binfmt interpreter for hppa
Setting /usr/bin/qemu-riscv32-static as binfmt interpreter for riscv32
Setting /usr/bin/qemu-riscv64-static as binfmt interpreter for riscv64
Setting /usr/bin/qemu-xtensa-static as binfmt interpreter for xtensa
Setting /usr/bin/qemu-xtensaeb-static as binfmt interpreter for xtensaeb
Setting /usr/bin/qemu-microblaze-static as binfmt interpreter for microblaze
Setting /usr/bin/qemu-microblazeel-static as binfmt interpreter for microblazeel
Setting /usr/bin/qemu-or1k-static as binfmt interpreter for or1k
Setting /usr/bin/qemu-hexagon-static as binfmt interpreter for hexagon
e[32;1m$ docker buildx create --platform linux/amd64,linux/arm64 --driver=docker-container --name=mybuilder --use --bootstrape[0;m
#1 [internal] booting buildkit
#1 pulling image moby/buildkit:buildx-stable-1
#1 pulling image moby/buildkit:buildx-stable-1 1.9s done
#1 creating container buildx_buildkit_mybuilder0
#1 creating container buildx_buildkit_mybuilder0 1.6s done
#1 DONE 3.5s
mybuilder
e[32;1m$ echo ""e[0;m

e[32;1m$ echo "Logging onto GitLab registry"e[0;m
Logging onto GitLab registry
e[32;1m$ docker login registry.gitlab.com -u "$GL_USERNAME" -p "$GL_TOKEN"e[0;m
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded
e[32;1m$ docker buildx lse[0;m
NAME/NODE    DRIVER/ENDPOINT   STATUS  BUILDKIT PLATFORMS
mybuilder *  docker-container                   
  mybuilder0 tcp://docker:2375 running v0.12.5  linux/amd64*, linux/arm64*, linux/amd64/v2, linux/amd64/v3, linux/riscv64, linux/ppc64, linux/ppc64le, linux/s390x, linux/386, linux/mips64le, linux/mips64, linux/arm/v7, linux/arm/v6
default      docker                             
  default    default           running v0.12.5  linux/amd64, linux/amd64/v2, linux/amd64/v3, linux/arm64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6, linux/ppc64, linux/mips64le, linux/mips64
e[32;1m$ docker context liste[0;m
NAME        DESCRIPTION                               DOCKER ENDPOINT     ERROR
default *   Current DOCKER_HOST based configuration   tcp://docker:2375   
Warning: DOCKER_HOST environment variable overrides the active context. To use a context, either set the global --context flag, or unset DOCKER_HOST environment variable.
e[32;1m$ docker --versione[0;m
Docker version 25.0.3, build 4debf41
e[32;1m$ cd $CI_PROJECT_DIRe[0;m
e[32;1m$ cd $CI_PROJECT_DIR/csbasee[0;m
e[32;1m$ sed -i "s+<tag_name>+$CS_IMAGES_VERSION+" test.dkre[0;m
e[32;1m$ docker buildx build --push --platform=linux/amd64,linux/arm64 -t "$CS_REPO/test2:$CS_IMAGES_VERSION" -f test.dkr --builder=mybuilder .e[0;m
#0 building with "mybuilder" instance using docker-container driver

#1 [internal] load build definition from test.dkr
#1 transferring dockerfile: 882B done
#1 DONE 0.0s

#2 [linux/amd64 internal] load metadata for docker.io/library/alpine:3.14
#2 DONE 0.9s

#3 [linux/arm64 internal] load metadata for docker.io/library/alpine:3.14
#3 DONE 0.9s

#4 [internal] load .dockerignore
#4 transferring context: 2B done
#4 DONE 0.0s

#5 [linux/amd64 1/2] FROM docker.io/library/alpine:3.14@sha256:0f2d5c38dd7a4f4f733e688e3a6733cb5ab1ac6e3cb4603a5dd564e5bfb80eed
#5 resolve docker.io/library/alpine:3.14@sha256:0f2d5c38dd7a4f4f733e688e3a6733cb5ab1ac6e3cb4603a5dd564e5bfb80eed 0.0s done
#5 DONE 0.1s

#6 [linux/arm64 1/2] FROM docker.io/library/alpine:3.14@sha256:0f2d5c38dd7a4f4f733e688e3a6733cb5ab1ac6e3cb4603a5dd564e5bfb80eed
#6 resolve docker.io/library/alpine:3.14@sha256:0f2d5c38dd7a4f4f733e688e3a6733cb5ab1ac6e3cb4603a5dd564e5bfb80eed 0.0s done
#6 sha256:422ed46b1a92579f7c475c0c19fade6880a8d98f23a2b4ccfb77c265d4f72dfc 2.73MB / 2.73MB 0.1s done
#6 extracting sha256:422ed46b1a92579f7c475c0c19fade6880a8d98f23a2b4ccfb77c265d4f72dfc 0.1s done
#6 DONE 0.3s

#5 [linux/amd64 1/2] FROM docker.io/library/alpine:3.14@sha256:0f2d5c38dd7a4f4f733e688e3a6733cb5ab1ac6e3cb4603a5dd564e5bfb80eed
#5 sha256:f7dab3ab2d6ec29aa28769bec35331fb485b5837501b1e8556413d8b5a79c9c8 2.83MB / 2.83MB 0.1s done
#5 extracting sha256:f7dab3ab2d6ec29aa28769bec35331fb485b5837501b1e8556413d8b5a79c9c8 0.1s done
#5 DONE 0.3s

#7 [linux/arm64 2/2] WORKDIR /home/hpcuser
#7 DONE 0.1s

#8 [linux/amd64 2/2] WORKDIR /home/hpcuser
#8 DONE 0.1s

#9 exporting to image
#9 exporting layers 0.1s done
#9 exporting manifest sha256:ec04d524b8d7639e3e5ecb6d9b22d559ca9bcdbad9f6edb4b825682280bf2346 0.0s done
#9 exporting config sha256:a369e285ba92cea82849c4b85ea387127ca523200c35d033d2c817cc5bb82377 0.0s done
#9 exporting attestation manifest sha256:4dfabe46cf8ed408a59fa0e95e9068772e38db77eacb24b8aff74b15c90afb46 0.0s done
#9 exporting manifest sha256:e376e409c64bad4f211ba504223948754930910be67651df657896bb0e784b64 0.0s done
#9 exporting config sha256:ced7250629481f42797b4cfa3981bda54be708966c9b5c06c9e9320035115a34 0.0s done
#9 exporting attestation manifest sha256:e0a47ac2bb0429710d17098f348efddb60702747511746b34b41512386d612b0 0.0s done
#9 exporting manifest list sha256:5df3dacae70ec0e6e26b2195687845a6b6681a7f0247a713ca6b9239b342cc6c
#9 ...

#10 [auth] txcorp/chemistream/chemistream-hpc-images/test2:pull,push token for registry.gitlab.com
#10 DONE 0.0s

#9 exporting to image
#9 exporting manifest list sha256:5df3dacae70ec0e6e26b2195687845a6b6681a7f0247a713ca6b9239b342cc6c 0.0s done
#9 pushing layers
#9 pushing layers 3.8s done
#9 pushing manifest for registry.gitlab.com/txcorp/chemistream/chemistream-hpc-images/test2:latest@sha256:5df3dacae70ec0e6e26b2195687845a6b6681a7f0247a713ca6b9239b342cc6c
#9 pushing manifest for registry.gitlab.com/txcorp/chemistream/chemistream-hpc-images/test2:latest@sha256:5df3dacae70ec0e6e26b2195687845a6b6681a7f0247a713ca6b9239b342cc6c 0.5s done
#9 DONE 4.6s
e[32;1m$ docker imagese[0;m
REPOSITORY                   TAG               IMAGE ID       CREATED         SIZE
moby/buildkit                buildx-stable-1   480495983c47   5 weeks ago     172MB
multiarch/qemu-user-static   latest            3539aaa87393   13 months ago   305MB
section_end:1710184757:step_script
e[0Ksection_start:1710184757:upload_artifacts_on_success
e[0Ke[0Ke[36;1mUploading artifacts for successful jobe[0;me[0;m
e[32;1mUploading artifacts...e[0;m
/builds/txcorp/chemistream/chemistream-hpc-images: found 1269 matching artifact files and directoriese[0;m 
.git: excluded 1 files                            e[0;m 
.git/**/*: excluded 19 files                      e[0;m 
e[0;33mWARNING: Upload request redirected                e[0;m  e[0;33mlocatione[0;m=https://gitlab.com/api/v4/jobs/6368170292/artifacts?artifact_format=zip&artifact_type=archive&expire_in=1+day e[0;33mnew-urle[0;m=https://gitlab.com
e[0;33mWARNING: Retrying...                              e[0;m  e[0;33mcontexte[0;m=artifacts-uploader e[0;33merrore[0;m=request redirected
Uploading artifacts as "archive" to coordinator... 201 Createde[0;m  ide[0;m=6368170292 responseStatuse[0;m=201 Created tokene[0;m=glcbt-65
section_end:1710184776:upload_artifacts_on_success
e[0Ksection_start:1710184776:cleanup_file_variables
e[0Ke[0Ke[36;1mCleaning up project directory and file based variablese[0;me[0;m
section_end:1710184777:cleanup_file_variables
e[0Ke[32;1mJob succeedede[0;m

I tried a few more things and looked at error logs and didnt find anything helpful. So I reluctantly decided to give up and return to this issue when gitlab docker enviornment on linux shared runners is as good as mac Docker Desktop.

So, I decided to build arm64 images on my mac (which has been working all along) and explicitly push them to the gitlab registry and tag them so I knew which ones to pull… relying on ‘caller logic’ to decide which image to pull. So I would (from my local mac Docker Desktop):

  • build arm64 image
  • tag arm64 image with gitlab registry name
  • push to gitlab registry

However this image appeared in my gitlab registry with 0 size!!! Moreover, I then removed my local docker arm64 image and pulled from my gitlab registry onto my local machine and it shows up with size 4.93GB as it should. So somehow, I think that the gitlab registry is not calculating the size correctly.

Does this sound like something that is possible? Now I’m beginning to think I was creating the images in my gitlab ci-cd pipelines correctly all along… but it wasnt showing with the correct size and I interpreted this as indicating it had built in error.

Any ideas?

1 Like

Hi there! I know this is a few weeks old, and hopefully you found some workaround in the meantime.

I wanted to throw my solution into the hat in case it helps someone – I spent many hours figuring out how to do multiplatform builds on shared gitlab runners. Was in the process of opening a ticket for a broken DinD shared runner and found this thread.

## docker image to build docker images 😵‍💫
FROM crazymax/docker:24.0.5 as docker

# override default tcp://docker:2375

ARG AWS_REGION
ENV AWS_REGION=${AWS_REGION:-us-east-2}

RUN apk add --no-cache --quiet \
        curl \
        xz \
        shadow \
        sudo \
        git \
        aws-cli \
        jq

COPY docker /docker
ENV PATH=$PATH:/docker/scripts

COPY scripts /scripts
ENV PATH=$PATH:/scripts

ENV DOCKER_HOST="" \
    DOCKER_HIDE_LEGACY_COMMANDS=1 \
    DOCKER_CLI_EXPERIMENTAL=enabled
docker buildx build --push --cache-from $DOCKER_TAG --target docker --builder=multiplatformbuilder --build-arg CI_JOB_TOKEN=$CI_JOB_TOKEN --platform linux/amd64,linux/arm64 --tag $DOCKER_TAG . >&2
echo "\n${DOCKER_TAG}\n" >&2
echo $DOCKER_TAG

run that on my macbook pro M1 and push it to the gitlab registry. from there, i build multi platform and arm-based images on amd64 shared runner.

# build-docker-images.sh

#!/bin/sh

set -e

echo "[$(date -Is)][$0] setting docker image build labels..." >&2

# Prepare image label options
# https://github.com/opencontainers/image-spec/blob/master/annotations.md
DOCKER_IMAGE_BUILD_LABEL_OPTIONS=" \
  --label org.opencontainers.image.source=$CI_PROJECT_URL \
  --label org.opencontainers.image.version=$CI_COMMIT_REF_NAME \
  --label org.opencontainers.image.revision=$CI_COMMIT_SHA \
  --label org.opencontainers.image.cicd=$GITLAB_CI \
  --label org.opencontainers.image.pipeline=$CI_JOB_URL \
  --label org.opencontainers.image.author=$(git config --get user.email || echo '') \
  --label org.opencontainers.image.created=$(date -u -Iseconds)"

  echo "[$(date -Is)][$0] \t$DOCKER_IMAGE_BUILD_LABEL_OPTIONS" >&2

echo "[$(date -Is)][$0] setting docker image cache sources..." >&2

# Prepare image cache options
DOCKER_IMAGE_BUILD_CACHE_SOURCES=" \
  --cache-from $CI_APPLICATION_REPOSITORY:$CI_APPLICATION_TAG \
  --cache-from $CI_APPLICATION_REPOSITORY:$DOCKER_IMAGE_BUILD_DEFAULT_TAG \
  $(echo $DOCKER_IMAGE_BUILD_TARGETS | xargs -n 1 | xargs -n 1 -I TARGET -r echo "--cache-from $CI_APPLICATION_REPOSITORY/TARGET:$CI_APPLICATION_TAG" | xargs)
  $(echo $DOCKER_IMAGE_BUILD_TARGETS | xargs -n 1 | xargs -n 1 -I TARGET -r echo "--cache-from $CI_APPLICATION_REPOSITORY/TARGET:$DOCKER_IMAGE_BUILD_DEFAULT_TAG" | xargs)"

echo "[$(date -Is)][$0] \t$DOCKER_IMAGE_BUILD_CACHE_SOURCES" >&2

DOCKER_IMAGE_BUILD_CACHE_DESTINATIONS=""
# DOCKER_IMAGE_BUILD_CACHE_DESTINATIONS=" \
#   --cache-to $CI_APPLICATION_REPOSITORY:$CI_APPLICATION_TAG \
#   --cache-to $CI_APPLICATION_REPOSITORY:$DOCKER_IMAGE_BUILD_DEFAULT_TAG \
#   $(echo $DOCKER_IMAGE_BUILD_TARGETS | xargs -n 1 | xargs -n 1 -I TARGET -r echo "--cache-to $CI_APPLICATION_REPOSITORY/TARGET:$CI_APPLICATION_TAG" | xargs)
#   $(echo $DOCKER_IMAGE_BUILD_TARGETS | xargs -n 1 | xargs -n 1 -I TARGET -r echo "--cache-to $CI_APPLICATION_REPOSITORY/TARGET:$DOCKER_IMAGE_BUILD_DEFAULT_TAG" | xargs)"

echo "[$(date -Is)][$0] \t$DOCKER_IMAGE_BUILD_CACHE_DESTINATIONS" >&2

build_image()
{
  echo "[$(date -Is)][$0] building $DOCKER_IMAGE_BUILD_DOCKERFILE..." >&2

  set -x
  docker build \
    $([ -n "$DOCKER_IMAGE_BUILD_NO_PUSH" ] && echo "" || echo "--push") \
    --file $DOCKER_IMAGE_BUILD_DOCKERFILE \
    --target "$DOCKER_IMAGE_BUILD_TARGET" \
    $DOCKER_IMAGE_BUILD_LABEL_OPTIONS \
    $([ -n "$DOCKER_IMAGE_BUILD_NO_CACHE" ] && echo "--no-cache" || echo "$DOCKER_IMAGE_BUILD_CACHE_SOURCES") \
    $([ -n "$DOCKER_IMAGE_BUILD_NO_CACHE" ] && echo "--no-cache" || echo "$DOCKER_IMAGE_BUILD_CACHE_DESTINATIONS") \
    --tag $DOCKER_IMAGE_BUILD_TAG \
    $DOCKER_IMAGE_BUILD_EXTRA_OPTIONS \
    $DOCKER_IMAGE_BUILD_EXTRA_BUILD_ARGS \
    "$DOCKER_IMAGE_BUILD_CONTEXT"

  set +x
}

echo "[$(date -Is)][$0] pushing $DOCKER_IMAGE_BUILD_TAG..." >&2

# Build and push the base image for the project
# DOCKER_IMAGE_BUILD_TARGET=""
DOCKER_IMAGE_BUILD_TAG="$CI_APPLICATION_REPOSITORY:$CI_APPLICATION_TAG"
build_image

echo "[$(date -Is)][$0] pushing other targets $DOCKER_IMAGE_BUILD_TARGETS..." >&2

# Build and push other image targets
DOCKER_IMAGE_BUILD_CACHE_SOURCES=""
echo "$DOCKER_IMAGE_BUILD_TARGETS" | xargs -n 1 -r | while read TARGET; do
  DOCKER_IMAGE_BUILD_TARGET="$TARGET"
  DOCKER_IMAGE_BUILD_TAG="$CI_APPLICATION_REPOSITORY/$TARGET:$CI_APPLICATION_TAG"
  build_image
done

for AWS ARM-based images, i set provenance to false for OCI image manifest compatibility

# .gitlab-ci.yml
...
my-ecr-image-builder-job:
  services:
    - docker:20.10.24-dind
  variables:
    DOCKER_DRIVER: overlay2
        DOCKER_IMAGE_BUILD_EXTRA_OPTIONS: --provenance=false --build-arg CI_JOB_TOKEN=$CI_JOB_TOKEN --platform linux/arm64 --build-arg BUILDKIT_INLINE_CACHE=1
  script:
    - # set all my CI_REGISTRY_IMAGE / $CI_APPLICATION_REPOSITORY / $CI_APPLICATION_TAG / etc. here
    - scripts/build-docker-images.sh
  tags:
    # use shared runners TODO use private EC2 runner
    - gitlab-org-docker

Thanks for the help. We are reworking our entire build design for our compute images to upgrade from Centos7 which is approaching EOL. I’ll attack this problem again once this reworking is done. We can close this thread for now. Thanks again for everyone’s time. Cheers,