GitLab container fails instantly (podman, aarch64)

The GitLab continer immediately fails on startup

Problem description

I have a Raspberry Pi 4 Model B (8-GB), on which I am running Fedora Server 36. I am using Podman as the container management system, instead of Docker.

The benefit to using Podman is the ability to have rootless containers with the default configuration. The daemon-less architecture is a plus. Helps me manage my containers using systemd :slight_smile:

OS information:

$ cat /etc/os-release

NAME="Fedora Linux"
VERSION="36 (Server Edition)"
ID=fedora
VERSION_ID=36
VERSION_CODENAME=""
PLATFORM_ID="platform:f36"
PRETTY_NAME="Fedora Linux 36 (Server Edition)"
ANSI_COLOR="0;38;2;60;110;180"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:36"
HOME_URL="https://fedoraproject.org/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/f36/system-administrators-guide/"
SUPPORT_URL="https://ask.fedoraproject.org/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=36
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=36
PRIVACY_POLICY_URL="https://fedoraproject.org/wiki/Legal:PrivacyPolicy"
VARIANT="Server Edition"
VARIANT_ID=server

Podman version:

$ podman --version

podman version 4.1.1

podman-compose version

$ podman-compose --version

['podman', '--version', '']
using podman version: 4.1.1
podman-composer version  1.0.3
podman --version
podman version 4.1.1
exit code: 0

gitlab-compose.yml:

version: 3.7

services:
    gitlab-web:
        image: gitlab/gitlab-ee:latest
        container_name: gitlab-govinda
        hostname: gitest.thefossguy.com
        restart: always
        environment:
            GITLAB_OMNIBUS_CONFIG: |
                external_url "https://gitest.thefossguy.com"
        ports:
            - "8050:80"
            - "8051:443"
            - "8052:22"
        volumes:
            - /trayimurti/containers/volumes/gitlab/config:/etc/gitlab:Z
            - /trayimurti/containers/volumes/gitlab/logs:/var/log/gitlab:Z
            - /trayimurti/containers/volumes/gitlab/data:/var/opt/gitlab:Z
        shm_size: '256m'

Open ports:

$ sudo firewall-cmd --list-ports

1313/tcp 8010/tcp 8011/tcp 8020/tcp 8030/tcp 8040/tcp 8050/tcp 8051/tcp 8052/tcp 8060/tcp 8061/tcp 8080/tcp 8443/tcp

File system:

$ tree /trayimurti/containers/volumes/gitlab/

/trayimurti/containers/volumes/gitlab/
├── config
├── data
└── logs

3 directories, 0 files

What are you seeing, and how does it differ from what you expect to see?

When I run podman-compose -f gitlab-compose.yml up, below is the output I get.

$ podman-compose -f gitlab-compose.yml up

['podman', '--version', '']
using podman version: 4.1.1
** excluding:  set()
['podman', 'network', 'exists', 'container-compose_default']
podman create --name=gitlab-govinda --label io.podman.compose.config-hash=123 --label io.podman.compose.project=container-compose --label io.podman.compose.version=0.0.1 --label com.docker.compose.project=container-compose --label com.docker.compose.project.working_dir=/home/pratham/my-git-repos/container-compose --label com.docker.compose.project.config_files=gitlab-compose.yml --label com.docker.compose.container-number=1 --label com.docker.compose.service=gitlab-web -e GITLAB_OMNIBUS_CONFIG=external_url "https://gitest.thefossguy.com"
 -v /trayimurti/containers/volumes/gitlab/config:/etc/gitlab:Z -v /trayimurti/containers/volumes/gitlab/logs:/var/log/gitlab:Z -v /trayimurti/containers/volumes/gitlab/data:/var/opt/gitlab:Z --net container-compose_default --network-alias gitlab-web -p 8050:80 -p 8051:443 -p 8052:22 --hostname gitest.thefossguy.com --shm-size 256m --restart always gitlab/gitlab-ee:latest
e6c08c5c99cf80d1b260db2235641e6d91e8c8f879a30ff4038746a4719ff32c
exit code: 0
podman start -a gitlab-govinda
{"msg":"exec container process `/assets/wrapper`: Exec format error","level":"error","time":"2022-07-27T17:27:09.000745663Z"}
exit code: 1

Upon starting the container, it immediately fails with the error message: {"msg":"exec container process '/assets/wrapper': Exec format error","level":"error","time":"2022-07-27T17:27:09.000745663Z"}.

Below is the complete log, nothing except for the same error message…

$ podman logs gitlab-govinda

{"msg":"exec container process `/assets/wrapper`: Exec format error","level":"error","time":"2022-07-27T17:49:14.000816230Z"}
{"msg":"exec container process `/assets/wrapper`: Exec format error","level":"error","time":"2022-07-27T17:49:15.000269738Z"}
{"msg":"exec container process `/assets/wrapper`: Exec format error","level":"error","time":"2022-07-27T17:49:15.000674320Z"}
{"msg":"exec container process `/assets/wrapper`: Exec format error","level":"error","time":"2022-07-27T17:49:16.000256608Z"}
{"msg":"exec container process `/assets/wrapper`: Exec format error","level":"error","time":"2022-07-27T17:49:24.000002119Z"}
{"msg":"exec container process `/assets/wrapper`: Exec format error","level":"error","time":"2022-07-27T17:49:24.000444004Z"}
{"msg":"exec container process `/assets/wrapper`: Exec format error","level":"error","time":"2022-07-27T17:49:25.000275618Z"}
{"msg":"exec container process `/assets/wrapper`: Exec format error","level":"error","time":"2022-07-27T17:49:25.000712005Z"}
{"msg":"exec container process `/assets/wrapper`: Exec format error","level":"error","time":"2022-07-27T17:49:26.000128775Z"}
{"msg":"exec container process `/assets/wrapper`: Exec format error","level":"error","time":"2022-07-27T17:49:26.000621771Z"}
{"msg":"exec container process `/assets/wrapper`: Exec format error","level":"error","time":"2022-07-27T17:49:27.000056762Z"}
{"msg":"exec container process `/assets/wrapper`: Exec format error","level":"error","time":"2022-07-27T17:49:27.000476499Z"}
{"msg":"exec container process `/assets/wrapper`: Exec format error","level":"error","time":"2022-07-27T17:49:27.000865988Z"}
{"msg":"exec container process `/assets/wrapper`: Exec format error","level":"error","time":"2022-07-27T17:49:28.000704415Z"}

I have not deployed a GitLab container using Docker, so I do not know what to expect (in the logs), but I expected that it at least runs up to the point that I can access the web GUI.

Can you link to any docs or other resources so we know where you have been?

My compose file is a direct copy of the one provided in the official Docs.

Since I Fedora has SELinux running in Enforcing mode, I am using the :Z SELinux label which Docker uses, and by extension Podman uses too.

What troubleshooting steps have you already taken?

I tried using the gitlab-ce image instead of gitlab-ee, but I get the same error in both cases.

Since the container instantly shuts down, there isn’t much that I can think of to troubleshoot this issue…

I have already verified that the ports I am using are open and the directories exist for the container to mount upon. I also own the domain thefossguy.com and intend to manage my instance of GitLab behind Caddy Server’s reverse proxy, which automatically manages SSL certs for subdomains. The caddy config already works for Gitea, but I am moving away since it has SSH-related issues.

Is this a case of PEBCAK?

Hi @pratham :wave: welcome to the GitLab community forum!

I think the problem here is a mismatch between the container images and your system architecture.

On DockerHub, gitlab/gitlab-ee:latest images are only available for the amd64 architecture. Docker Hub There are no official gitlab/gitlab-ee images built for aarch64 or arm64

To run GitLab on a Raspberry Pi running Fedora 36, I suggest downloading the latest gitlab-ee-15.2.x-ee.0.el8.aarch64.rpm package from gitlab/gitlab-ee - Packages · packages.gitlab.com and installing that with dnf install gitlab-ee-15.2.x-ee.0.el8.aarch64.rpm

1 Like

Ouch, I didn’t check that GitLab was offered in the arm64 architecture or not. So podman is trying to run an x86 image on arm64.

Are there any plans to provide an arm64 image anytime soon?

So podman is trying to run an x86 image on arm64.

Exactly!

Are there any plans to provide an arm64 image anytime soon?

We have an open issue about this here:

Until an official arm64 image is released, I’d use the GitLab Linux package.

Alternatively, you could build your own image or use a third-party arm64 image of GitLab CE like Docker .

1 Like

I would assume Podman would only pull images of native arch.

Thanks, will keep an eye on it!

1 Like