GitLab CI Can't Run Local Binary File on Debian Bullseye Pi Runner

GitLab CI Can’t Run Local Binary File?

I’m facing a weird issue where on the pipeline throughput, it shows that the downloaded binary program is placed at the right location:

$ ls -la "${PWD}/${MONTEUR_FS}"
total 6580
drwxr-xr-x 2 root root    4096 Jan  7 10:27 .
drwxr-xr-x 3 root root    4096 Jan  7 10:27 ..
-rwxr-xr-x 1 root root 6727312 Jan  5 13:18 monteur

As I export the PATH and the directory, it could not locate the monteur binary program at all:

$ export PATH="PATH:{$PWD}/${MONTEUR_FS}/monteur"
$ monteur help
/bin/bash: line 178: monteur: command not found
Cleaning up project directory and file based variables
00:03
ERROR: Job failed: exit code 1

I’m using a simple Raspberry Pi runner. Last .gitlab-ci.yml is:

image: debian:latest

stages:
    - test
    - build
    - docs

variables:
    TERM: "xterm"
    GITLAB_CI: "true"
    MONTEUR_VERSION: "v0-0-2"
    MONTEUR_OS: "linux"
    MONTEUR_FS: ".monteurFS/bin"

test:
    stage: test
    tags:
        - linux
    environment:
        name: production
    except:
        refs:
            - gh-pages
    interruptible: true
    coverage: '/total:\s{1,}\(statements\)\s{1,}(\d+.\d+%)/'
    variables:
        MONTEUR_URL: "https://monteur.zoralab.com/releases/archives"
        MONTEUR_CHECKSUM: "checksum-sha512.txt"
    before_script:
        - apt-get update -y
        - apt-get upgrade -y
        - |
          apt-get install \
            wget \
          -y
        - MONTEUR_ARCH="$(dpkg --print-architecture)"
        - |
          case "$MONTEUR_ARCH" in
          armhf)
              MONTEUR_ARCH=arm
              ;;
          aarch64)
              MONTEUR_ARCH=arm64
              ;;
          esac
        - MONTEUR_FILE="monteur-${MONTEUR_VERSION}-${MONTEUR_OS}-${MONTEUR_ARCH}.tar.gz"
        - mkdir -p tmp && cd tmp
        - wget -O "$MONTEUR_FILE" "${MONTEUR_URL}/${MONTEUR_VERSION}/${MONTEUR_FILE}"
        - wget -O "$MONTEUR_CHECKSUM" "${MONTEUR_URL}/${MONTEUR_VERSION}/${MONTEUR_CHECKSUM}"
        - sha512sum --ignore-missing -c "$MONTEUR_CHECKSUM"
        - tar -xf "$MONTEUR_FILE"
        - chmod +x monteur
        - chown root:root monteur
        - cd ..
        - mkdir -p "$MONTEUR_FS"
        - mv ./tmp/monteur "${MONTEUR_FS}/."
        - rm -rf tmp/
        - ls -la "${PWD}/${MONTEUR_FS}"
        - export PATH="PATH:{$PWD}/${MONTEUR_FS}/monteur"
        - monteur help
    script:
        - echo "pending test"

build:
    stage: build
    tags:
        - linux
    environment:
        name: production
    except:
        refs:
            - gh-pages
    environment:
        name: production
    script:
        - echo "pending build"

compose:
    stage: docs
    tags:
        - linux
    environment:
        name: production
    except:
        refs:
            - gh-pages
    only:
        refs:
            - schedules
    script:
        - echo "pending compose"

pages:
    stage: docs
    tags:
        - linux
    environment:
        name: production
    only:
        refs:
            - gh-pages
    artifacts:
        paths:
            - public
        expire_in: 1 day
    script:
        - mkdir -p public
        - shopt -s extglob
        - mv !(public|.*) public

I had tried every attempts that I know with BASH:

  1. Export PATH
  2. Export monteur with full path.
  3. Place under /usr/bin.
  4. Place under /usr/local/bin.

None of them works (all of them complaint that monteur is missing despite ls is able to detect the location). If I execute locally, everything is fine.

I’m on gitlab.com. The project is open-source and the developing pipeline is available at: test (#1951444920) · Jobs · ZORALab / Monteur · GitLab.

On the same runner system downloading the same file, it works:

I’m out of ideas. If anyone has some clues, please shed some light. Thanks!


Update: attempted image entrypoints (Run your CI/CD jobs in Docker containers | GitLab):

  1. Empty - failed (test (#1951724188) · Jobs · ZORALab / Monteur · GitLab)
  2. /bin/bash, -c - failed (test (#1951724188) · Jobs · ZORALab / Monteur · GitLab)
  3. /bin/bash, -l, -c - failed (test (#1951829673) · Jobs · ZORALab / Monteur · GitLab)
  4. "" - failed (test (#1951866442) · Jobs · ZORALab / Monteur · GitLab)

Attempted CMD workaround:

  1. exec monteur - failed (test (#1951910207) · Jobs · ZORALab / Monteur · GitLab)

Update Jan 8, 2022 - I swapped the runners and I believe @balonik tested it on amd64 that works flawlessly (test (#1953807606) · Jobs · ZORALab / Monteur · GitLab). Hence, I strongly believe this issue is strictly related to docker on Raspberry Pi Runner.

Hi @Holloway ,
you have 2 typos in that line, first a missing $ in PATH: and 2nd badly position curly bracket in {$PWD} so you are basically screwing your PATH env variable.
It should be - export PATH="${PATH}:${PWD}/${MONTEUR_FS}/monteur"

1 Like

Thanks for spotting the typo. Amended. :grin: However, I’m still facing the missing binary executable problem.

I’m been trying to debug the problem for quite some times before that line, right after the untar:


$ sha512sum --ignore-missing -c "$MONTEUR_CHECKSUM"
monteur-v0-0-2-linux-arm.tar.gz: OK
$ tar -xf "$MONTEUR_FILE"
$ chmod +x monteur
$ chown root:root monteur
$ ls -lad "$PWD"/*
-rw-r--r-- 1 1000 1000   78407 Dec 24 00:56 /builds/zoralab/monteur/tmp/License.pdf
-rw-r--r-- 1 root root    1138 Jan  7 04:16 /builds/zoralab/monteur/tmp/checksum-sha512.txt
-rwxr-xr-x 1 root root 6727312 Jan  5 13:18 /builds/zoralab/monteur/tmp/monteur
-rw-r--r-- 1 root root 2781958 Jan  7 04:16 /builds/zoralab/monteur/tmp/monteur-v0-0-2-linux-arm.tar.gz
$ exec ./monteur help
/bin/bash: line 168: /builds/zoralab/monteur/tmp/monteur: No such file or directory

Alright, so looking at the commands the correct line is export PATH="${PATH}:${PWD}/${MONTEUR_FS}"
Without the /monteur at the end since you need to put paths to directories there, not binaries.

I’m gonna call off the day. Last attempt was: test (#1952018197) · Jobs · ZORALab / Monteur · GitLab

Yeah, not a concern for now, I’m left with these possibilities:

  1. Some kind of configurations either with docker or gitlab runner.
  2. Something went wrong with debian:latest docker image at the moment.

I can’t debug further the problem without some lead when ls is able to locate the problem but somehow, the scripted shell script by gitlab-runner can’t locate it, even with overwritten entry-point.

I have tested it just replacing the export PATH line with export PATH="${PATH}:${PWD}/${MONTEUR_FS}" and it works like a charm.

No it doesn’t (test (#1953681963) · Jobs · ZORALab / Monteur · GitLab)

$ rm -rf tmp/
$ ls -lad "${PWD}/${MONTEUR_BIN}"/*
-rwxr-xr-x 1 root root 6727312 Jan  5 13:18 /builds/zoralab/monteur/.monteurFS/bin/monteur
$ PATH="${PATH}:${PWD}/${MONTEUR_BIN}"
$ echo "$PATH"
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/builds/zoralab/monteur/.monteurFS/bin
$ monteur help
/bin/bash: line 180: /builds/zoralab/monteur/.monteurFS/bin/monteur: No such file or directory
Cleaning up project directory and file based variables
00:04
ERROR: Job failed: exit code 1

It’s entirely a different issue. Note that I can operate towards the binary file: test (#1953723137) · Jobs · ZORALab / Monteur · GitLab

        - echo "$PATH"
        - cat "${PWD}/${MONTEUR_BIN}/monteur"

I swapped the runners and I believe @balonik tested it on amd64 that works flawlessly (test (#1953807606) · Jobs · ZORALab / Monteur · GitLab). Hence, I strongly believe this issue is strictly related to docker on Raspberry Pi Runner.

Re-titled to reflect the latest findings.

Okay, here is the findings:

Thanks @balonik for spotting the PATH typo. This definitely fixed the problem for amd64 type GitLab runner (The final result is test (#1954133763) · Jobs · ZORALab / Monteur · GitLab).

However, with the same GitLab CI job, the Go binary application did not run at all on Raspberry Pi docker’s debian:latest image (test (#1954108724) · Jobs · ZORALab / Monteur · GitLab).

Note that both test results above used apt install method.

The same binary application was SSH-ed into the Raspberry Pi and tested out manually which shown that it works. Hence, this rules out the binary was improperly built.

That being said: I’m highly suspecting the root cause is related to the binary interpretation (LE/BE, some Raspberry Pi low-level tweaks that are not present on debian:latest image, etc), either caused by:

  1. Frankendebianed by Raspbian team to the point of catastrophic deviation from mainline Debian.
  2. Debian image was improperly prepared.

My bet is on cause 1 since the file is actually readable (https://storage.googleapis.com/gitlab-gprd-artifacts/36/51/36517ac0b81cf6df2c0aaef383f042c2fd5d5cc50e2c1be9a591317455652d1f/2022_01_08/1953953354/2118594341/job.log?response-content-type=text%2Fplain%3B%20charset%3Dutf-8&response-content-disposition=inline&GoogleAccessId=gitlab-object-storage-prd@gitlab-production.iam.gserviceaccount.com&Signature=bKtuCdWkIrN0fpuYYjue9AuRTP551ZFbLMpc%2B%2BCIHELB94aHL4b0hlnii6Rd xy%2Fpjs6N1msFMibX3HEdXZ0Fvvjl0hdaYo4NW8hLA%2FvgwDRqOBfCD%2Fzd5DjC M5aqyg8cQpCfollvDqK%2BvswTRgWzz9DRicrskTxxXDckaigonAdzNyt67LOW 3ZwatMzF%2BlRi67hArA3MSRGGRxd78nKTnF2QbuXMXaadgrW23M4qiowhTbEY 1BXHQ9pSgb59f1eRE81fPQJzeY2KXwPUUoBxNwp%2FV2hM9l02jKfPy5Wi9Bn7 Up0Grj0CdNiDXFkXBJwZHVqqJhzRt%2BJOOc2RqlH2Fw%3D%3D&Expires=1641608323) and Raspberry Pi is a known and popular FrankenDebian.


Solution can be any of the following:

  1. Switch the raspberry pi gitlab-runner not to use Docker image to something like Shell executors.
  2. Build custom docker image (no guarentee it will work).
  3. Don’t use raspberry pi for runner. There are other lower-power ARM64 single board computer options available.