GitLab Runner with Podman Socket

How to Use GitLab Self-managed
, , ,
1

Hello everyone

I hope you are doing well!

I have set up a GitLab runner on a Linux Debian machine via Podman. Unfortunately, the GitLab runner crashes every 100 builds as far as I can see. Why is that? My GitLab pipelines fail all the time. This produces a lot of noise.

my config.toml:

concurrent = 100
check_interval = 0
connection_max_age = "15m0s"
shutdown_timeout = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "self-hosted-runner"
  url = "https://gitlab.com"
  id = 47105945
  token = "glrt-[REDACTED]"
  token_obtained_at = 2025-05-06T13:44:13Z
  token_expires_at = 0001-01-01T00:00:00Z
  executor = "docker"
  environment = ["DOCKER_AUTH_CONFIG={\"auths\":{\"registry.gitlab.com\":{\"auth\":\"token=\"}}}"]
  [runners.cache]
    MaxUploadedArchiveSize = 0
    [runners.cache.s3]
    [runners.cache.gcs]
    [runners.cache.azure]
  [runners.docker]
    host = "unix:///var/run/podman.sock"
    tls_verify = false
    image = "docker.io/library/debian:latest"
    privileged = false
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/cache"]
    pull_policy = ["if-not-present"]
    shm_size = 0
    network_mtu = 0

my compose.yml looks like this:

services:
  gitlab-runner:
    image: docker.io/gitlab/gitlab-runner:v17.11.0
    container_name: runner-01
    volumes:
      - /home/podman/runner-01/config:/etc/gitlab-runner
      - /run/user/1000/podman/podman.sock:/var/run/podman.sock
    environment:
      - DOCKER_TLS_CERTDIR=
      - DOCKER_HOST=unix:///var/run/podman.sock
    networks:
      - gitlab-network
    restart: always

networks:
  gitlab-network:
    name: prod-gl-runner
    external: true

The logs look like this immediately before the crash:

WARNING: Failed to exec create to container: Error response from daemon: can only create exec sessions on running containers: container state improper (docker.go:1463:0s)  error=Error response from daemon: can only create exec sessions on running containers: container state improper (docker.go:1463:0s) runner=cMl34-FMs
Job succeeded                                       duration_s=-1e-09 job=10060842581 project=1348937429 runner=cMl34-FMs
Appending trace to coordinator...ok                 code=202 job=10060842581 job-log=0-25843 job-status=running runner=cMl34-FMs sent-log=22994-25842 status=202 Accepted update-interval=1m0s
Updating job...                                     bytesize=25843 checksum=crc32:424d6266 job=10060842581 runner=cMl34-FMs
Submitting job to coordinator...accepted, but not yet completed  bytesize=25843 checksum=crc32:424d6266 code=202 job=10060842581 job-status=running runner=cMl34-FMs update-interval=1s
Updating job...                                     bytesize=25843 checksum=crc32:424d6266 job=10060842581 runner=cMl34-FMs
Submitting job to coordinator...ok                  bytesize=25843 checksum=crc32:424d6266 code=200 job=10060842581 job-status=success runner=cMl34-FMs update-interval=0s
Removed job from processing list                    builds=0 job=10060842581 max_builds=100 project=1348937429 repo_url=https://gitlab.com/path/to/repo.git time_in_queue_seconds=3
WARNING: [runWait] received stop signal             builds=0 max_builds=100 stop-signal=terminated
WARNING: Graceful shutdown not finished properly. To gracefully clean up running plugins please use SIGQUIT (ctrl-\) instead of SIGINT (ctrl-c)  builds=0 error=received stop signal: terminated max_builds=100
WARNING: Starting forceful shutdown                 StopSignal=terminated builds=0 max_builds=100 shutdown-timeout=30s
WARNING: Checking for jobs... failed                runner=b-WFFRZZq status=couldn't execute POST against https://gitlab.com/api/v4/jobs/request: Post "https://gitlab.com/api/v4/jobs/request": context canceled
WARNING: Checking for jobs... failed                runner=V6n6SISTn status=couldn't execute POST against https://gitlab.com/api/v4/jobs/request: Post "https://gitlab.com/api/v4/jobs/request": context canceled
WARNING: Checking for jobs... failed                runner=cMl34-FMs status=couldn't execute POST against https://gitlab.com/api/v4/jobs/request: Post "https://gitlab.com/api/v4/jobs/request": context canceled
All workers stopped.                                builds=0 max_builds=100
Shutting down executor providers                    builds=0 max_builds=100 shutdown-timeout=30s
WARNING: [handleForcefulShutdown] received stop signal  builds=0 max_builds=100 stop-signal=terminated
WARNING: Forceful shutdown not finished properly    builds=0 error=forced exit with stop signal: terminated max_builds=100
FATAL: Service run failed                           error=forced exit with stop signal: terminated
Runtime platform                                    arch=amd64 os=linux pid=2 revision=0f67ff19 version=17.11.0
Starting multi-runner from /etc/gitlab-runner/config.toml...  builds=0 max_builds=0
Running in system-mode

The podman socket is activated and located under:

podman@prod-00:~$ ls /run/user/1000/podman/podman.sock
/run/user/1000/podman/podman.sock

The socket is running probably:

podman@prod-00:~$ systemctl status podman.socket
● podman.socket - Podman API Socket
     Loaded: loaded (/lib/systemd/system/podman.socket; enabled; preset: enabled)
     Active: active (listening) since Tue 2025-05-06 15:37:34 CEST; 1 week 3 days ago
   Triggers: ● podman.service
       Docs: man:podman-system-service(1)
     Listen: /run/podman/podman.sock (Stream)
     CGroup: /system.slice/podman.socket

I hope someone can help me with this! Thanks in advance :))

Versions

Please add an x whether options apply, and add the version information.

  • Self-managed
  • GitLab.com SaaS
  • Dedicated

Versions

  • GitLab (Web: /help or self-managed system information sudo gitlab-rake gitlab:env:info):
2

Is podman already supported? Because this is also a strange error:

WARNING: Failed to exec create to container: Error response from daemon: can only create exec sessions on running containers: container state improper (docker.go:1463:0s)  error=Error response from daemon: can only create exec sessions on running containers: container state improper (docker.go:1463:0s) runner=cMl34-FMs

Can someone please help me?

3

From a quick forum search: Gitlab runner setup with podman some of the stuff there should help.

4

Thank you for your reply @iwalker

Unfortunately this does not solve the problem. I have done everything as described in the previous forum posts, but the problem still persists

5

Hi, I am using podman to run runner and it works. I use it with this config

  [runners.docker]
    host = "unix:///run/user/<id_of_user_running_podman_socket>/podman/podman.sock"
    tls_verify = false
    image = "registry/proxy-quay/buildah/stable:latest"
    privileged = true

But I don’t use compose.

6

Having the same issue with podman; the container lives for <0,1s and dies, as seen in the podman journal:
2025-10-28 22:36:12.767713306 +0100 CET m=+4.598666873 container start
2025-10-28 22:36:12.792966009 +0100 CET m=+4.623919581 container died

So if I only perform some simple tasks like mkdir or ls, the pipeline succeeds, so the container is used while it lives.

I’m running the gitlab-runner as user with settings like @mesenger posted, tried both privileged = true/false, doesn’t make a difference.

Anyone has an idea how to get more debug output to get to the root of this?