Docker executor with systemd

One of my projects requires systemd to run in the background.
This is perfectly fine for use when installed on the system.
But for testing with gitlab and a docker executor, this is bad.

I can run a docker container with systemd installed.
The entrypoint has to be “/lib/systemd/systemd” (it needs to have PID 1).
Systemd starts and gets to a login shell. I can bypass this and
automatically login to the system.
So “docker run …” gives me a shell in the container.

However, gitlab passes a script to the container upon start.
That would make that script to be PID 1 which does not work with systemd.

Is there another way to get this to work?

I would like to do the following:
(1) “docker run” starts the container, and
(2) “docker exec” runs the script.

Thanks for taking the time to be thorough in your request, it really helps! :blush:*

1 Like

Hi @Dennis_1 :wave:

I’ve encountered this same problem before and I couldn’t figure out any way to get systemd to work with GitLab Runner Docker executor. If anyone knows how to do this, please share! :pray:

A potential workaround could be using GitLab Runner’s Shell executor on a machine with Docker installed for any CI jobs that require systemd and docker.

1 Like

I believe @nicolamori managed to solve this see this post on stackoverflow, however my attempt to duplicate it initially resulted in a Docker container which, when run on a non-privileged runner, simply returned ERROR: Job failed: exit code 255, however when run on a privileged runner it works perfectly on Debian Bookworm and Ubuntu Jammy however with Debian Bullseye the CI job doesn’t terminate until the GitLab CI timeout is reached — however this might be a molecule issue?

The only part of this potential solution I’ve written myself is the last section of the Dockerfile:

COPY entrypoint.sh /entrypoint.sh
COPY bash.service /etc/systemd/system/bash.service
RUN chown root:root /entrypoint.sh \
    && chmod 755 /entrypoint.sh \
    && chown root:root /etc/systemd/system/bash.service \
    && chmod 644 /etc/systemd/system/bash.service \
    && systemctl enable bash.service
ENTRYPOINT ["/entrypoint.sh"]

Everything else has been copied from the stackoverflow comment linked above.

The only additional Debian package I have installed in the container, for this (I’m doing other things like installing Ansible) is systemd.

Note that for Debian Bullseye systemd is at /lib/systemd/systemd rather than /usr/lib/systemd/systemd where it is found for Debian Bookworm and Ubuntu Jammy.

One potential improvement that the entrypoint.sh script could do with is a check if the container is running in privileged mode with a error message being returned if it is not, see for example the suggestion here however this might not be possible as the check might interfere with the PID assignment?

1 Like

I wonder if this method for running systemd in a non-privileged Docker container could be used with GitLab?

This solution only works with GitLab runner up to version 16.5.0, version 16.6.0 breaks it, if anyone has a solution to this I’d be interested to hear it!

UPDATE: This has been fixed with version 16.6.1.

Hello,
thank you to post these suggestions very useful to address gitlab-runners in docker systemd.

I tried this solution, but CI job doesn’t terminate until the GitLab CI timeout is reached. This is the stalled state:

● runner-tavcf519-project-18162-concurrent-1
    State: stopping
     Jobs: 0 queued
   Failed: 1 units
    Since: Thu 2023-12-14 11:18:24 UTC; 7min ago
   CGroup: /docker/0c9a9c030b3db878465480357dedbbc79891f31e7e592bad64fa596f9e76cee5
           ├─4195 bash
           ├─4213 systemctl status
           ├─4214 less
           ├─init.scope
           │ ├─1 /usr/lib/systemd/systemd
           │ └─7 /usr/bin/coreutils --coreutils-prog-shebang=sleep /usr/bin/sleep infinity
           └─system.slice
             └─systemd-journald.service
               └─19 /usr/lib/systemd/systemd-journald
# systemctl list-units --failed
  UNIT                 LOAD   ACTIVE SUB    DESCRIPTION
● systemd-exit.service loaded failed failed Exit the Session
...
# journalctl -u systemd-exit.service
-- Logs begin at Thu 2023-12-14 11:18:24 UTC, end at Thu 2023-12-14 11:25:05 UTC. --
Dec 14 11:25:05 runner-tavcf519-project-18162-concurrent-1 systemd[1]: systemd-exit.service: Succeeded.
Dec 14 11:25:05 runner-tavcf519-project-18162-concurrent-1 systemd[1]: Started Exit the Session.
Dec 14 11:25:05 runner-tavcf519-project-18162-concurrent-1 systemd[1]: systemd-exit.service: Succeeded.
Dec 14 11:25:05 runner-tavcf519-project-18162-concurrent-1 systemd[1]: Started Exit the Session.
Dec 14 11:25:05 runner-tavcf519-project-18162-concurrent-1 systemd[1]: systemd-exit.service: Succeeded.
Dec 14 11:25:05 runner-tavcf519-project-18162-concurrent-1 systemd[1]: Started Exit the Session.
Dec 14 11:25:05 runner-tavcf519-project-18162-concurrent-1 systemd[1]: systemd-exit.service: Succeeded.
Dec 14 11:25:05 runner-tavcf519-project-18162-concurrent-1 systemd[1]: Started Exit the Session.
Dec 14 11:25:05 runner-tavcf519-project-18162-concurrent-1 systemd[1]: systemd-exit.service: Succeeded.
Dec 14 11:25:05 runner-tavcf519-project-18162-concurrent-1 systemd[1]: Started Exit the Session.
Dec 14 11:25:05 runner-tavcf519-project-18162-concurrent-1 systemd[1]: systemd-exit.service: Start request repeated too quickly.
Dec 14 11:25:05 runner-tavcf519-project-18162-concurrent-1 systemd[1]: systemd-exit.service: Failed with result 'start-limit-hit'.
Dec 14 11:25:05 runner-tavcf519-project-18162-concurrent-1 systemd[1]: Failed to start Exit the Session.

I can’t understand how to fix this. I don’t run molecule tests.

I’ve found a good alternative for run command in systemd here, but it doesn’t work in gitlab-runners because it requires tty.

Thank you very much for every hints
Marco

1 Like

I have found that with Debian Bullseye the jobs never complete, I have raised this on molecule forum, on Debian Bookworm, Debian Trixie and Ubuntu Jammy there isn’t this problem, I have assumed that this might be to do with the systemd version but this is just a guess, I have been working around this by not testing using Bullseye, which isn’t ideal but I couldn’t find a soultion to this problem…

I think to have fixed this by adding --force to systemctl exit in bash.service:

ExecStopPost=/usr/bin/systemctl exit --force $EXIT_STATUS

I don’t know if it’s a robust solution, it seems it depends on system you have… anyway it fork for me.

1 Like

Thanks, I’ll try that for Bullseye, incidently was there a reason to omit --system?

I really don’ know, I think it’s useless. The systemctl exit man page says

This is only supported for user service managers (i.e. in conjunction with the --user option) or in containers and is equivalent to poweroff otherwise.

1 Like