Arm64 emulation segfaults on SaaS Runners but not on Self Hosted ones, why?


My goal is creating a rootfs for arm64 based embedded device using CI/CD pipeline. In order to keep flexibility, I am trying to utilize docker executor not the shell. I have an access to a self-hosted Gitlab instance configured with a docker executor. Since I am not the maintainer of the self-hosted instance, I don’t know full details of self-hosted instance.

The first thing I tried is running an arm64 Ubuntu on Gitlab SaaS Runner (which is a docker executor fundamentally). This is the .gitlab-ci.yml:

  image: docker
    - docker:dind
    - docker run --privileged --rm tonistiigi/binfmt --install arm64 # looks like redundant, arm64 emulation is already supported.
    - docker run --rm arm64v8/ubuntu bash -c "apt-get -y update; apt-get -y install sudo; apt-get -y upgrade"
    #- docker run --rm arm64v8/ubuntu ldconfig --version # hangs

Here is the public repo with this CI/CD configuration and the run result:

I traced the problem and found ldconfig in Ubuntu rootfs gives segmentation error. There are multiple issues over the Internet about this problem with QEMU emulation. The last line on CI/CD flow executes ldconfig --version. This line is executed on the self-hosted Gitlab instance/runner perfectly but not on SaaS runner.

Question: Why does this run fail on Gitlab Saas runners but not on self hosted instance?

This should be related with QEMU and differences between virtualization stack of two runners. Our runner is running gitkab-runner 15.9.1 but the Saas ones run 15.9.0~beta.115.g598a7c91, slightly older version. Can it be related?

Question: How do runners emulate arm64 natively?

Even if my CI/CD flow include docker run --privileged --rm tonistiigi/binfmt --install arm64 step, removing this step doesn’t cause any problem. It also says that * arm64 qemu-aarch64 already registered*. Is arm64 emulation natively supported on runners? Is it related to Docker buildx?

Thank you.

Update Actually no need to use dind, the following configuration can be used as MRE:

  image: arm64v8/ubuntu
    - uname -a
    - apt-get -y update
    - apt-get -y install sudo
    - apt-get -y upgrade