One of my projects requires systemd to run in the background.
This is perfectly fine for use when installed on the system.
But for testing with gitlab and a docker executor, this is bad.
I can run a docker container with systemd installed.
The entrypoint has to be “/lib/systemd/systemd” (it needs to have PID 1).
Systemd starts and gets to a login shell. I can bypass this and
automatically login to the system.
So “docker run …” gives me a shell in the container.
However, gitlab passes a script to the container upon start.
That would make that script to be PID 1 which does not work with systemd.
Is there another way to get this to work?
I would like to do the following:
(1) “docker run” starts the container, and
(2) “docker exec” runs the script.
Thanks for taking the time to be thorough in your request, it really helps! *
I’ve encountered this same problem before and I couldn’t figure out any way to get systemd to work with GitLab Runner Docker executor. If anyone knows how to do this, please share!
A potential workaround could be using GitLab Runner’s Shell executor on a machine with Docker installed for any CI jobs that require systemd and docker.
I believe @nicolamori managed to solve this see this post on stackoverflow, however my attempt to duplicate it initially resulted in a Docker container which, when run on a non-privileged runner, simply returned ERROR: Job failed: exit code 255, however when run on a privileged runner it works perfectly on Debian Bookworm and Ubuntu Jammy however with Debian Bullseye the CI job doesn’t terminate until the GitLab CI timeout is reached — however this might be a molecule issue?
The only part of this potential solution I’ve written myself is the last section of the Dockerfile:
Everything else has been copied from the stackoverflow comment linked above.
The only additional Debian package I have installed in the container, for this (I’m doing other things like installing Ansible) is systemd.
Note that for Debian Bullseye systemd is at /lib/systemd/systemd rather than /usr/lib/systemd/systemd where it is found for Debian Bookworm and Ubuntu Jammy.
One potential improvement that the entrypoint.sh script could do with is a check if the container is running in privileged mode with a error message being returned if it is not, see for example the suggestion here however this might not be possible as the check might interfere with the PID assignment?
This solution only works with GitLab runner up to version 16.5.0, version 16.6.0 breaks it, if anyone has a solution to this I’d be interested to hear it!
Hello,
thank you to post these suggestions very useful to address gitlab-runners in docker systemd.
I tried this solution, but CI job doesn’t terminate until the GitLab CI timeout is reached. This is the stalled state:
● runner-tavcf519-project-18162-concurrent-1
State: stopping
Jobs: 0 queued
Failed: 1 units
Since: Thu 2023-12-14 11:18:24 UTC; 7min ago
CGroup: /docker/0c9a9c030b3db878465480357dedbbc79891f31e7e592bad64fa596f9e76cee5
├─4195 bash
├─4213 systemctl status
├─4214 less
├─init.scope
│ ├─1 /usr/lib/systemd/systemd
│ └─7 /usr/bin/coreutils --coreutils-prog-shebang=sleep /usr/bin/sleep infinity
└─system.slice
└─systemd-journald.service
└─19 /usr/lib/systemd/systemd-journald
# systemctl list-units --failed
UNIT LOAD ACTIVE SUB DESCRIPTION
● systemd-exit.service loaded failed failed Exit the Session
...
# journalctl -u systemd-exit.service
-- Logs begin at Thu 2023-12-14 11:18:24 UTC, end at Thu 2023-12-14 11:25:05 UTC. --
Dec 14 11:25:05 runner-tavcf519-project-18162-concurrent-1 systemd[1]: systemd-exit.service: Succeeded.
Dec 14 11:25:05 runner-tavcf519-project-18162-concurrent-1 systemd[1]: Started Exit the Session.
Dec 14 11:25:05 runner-tavcf519-project-18162-concurrent-1 systemd[1]: systemd-exit.service: Succeeded.
Dec 14 11:25:05 runner-tavcf519-project-18162-concurrent-1 systemd[1]: Started Exit the Session.
Dec 14 11:25:05 runner-tavcf519-project-18162-concurrent-1 systemd[1]: systemd-exit.service: Succeeded.
Dec 14 11:25:05 runner-tavcf519-project-18162-concurrent-1 systemd[1]: Started Exit the Session.
Dec 14 11:25:05 runner-tavcf519-project-18162-concurrent-1 systemd[1]: systemd-exit.service: Succeeded.
Dec 14 11:25:05 runner-tavcf519-project-18162-concurrent-1 systemd[1]: Started Exit the Session.
Dec 14 11:25:05 runner-tavcf519-project-18162-concurrent-1 systemd[1]: systemd-exit.service: Succeeded.
Dec 14 11:25:05 runner-tavcf519-project-18162-concurrent-1 systemd[1]: Started Exit the Session.
Dec 14 11:25:05 runner-tavcf519-project-18162-concurrent-1 systemd[1]: systemd-exit.service: Start request repeated too quickly.
Dec 14 11:25:05 runner-tavcf519-project-18162-concurrent-1 systemd[1]: systemd-exit.service: Failed with result 'start-limit-hit'.
Dec 14 11:25:05 runner-tavcf519-project-18162-concurrent-1 systemd[1]: Failed to start Exit the Session.
I can’t understand how to fix this. I don’t run molecule tests.
I’ve found a good alternative for run command in systemd here, but it doesn’t work in gitlab-runners because it requires tty.
I have found that with Debian Bullseye the jobs never complete, I have raised this on molecule forum, on Debian Bookworm, Debian Trixie and Ubuntu Jammy there isn’t this problem, I have assumed that this might be to do with the systemd version but this is just a guess, I have been working around this by not testing using Bullseye, which isn’t ideal but I couldn’t find a soultion to this problem…
I really don’ know, I think it’s useless. The systemctl exit man page says
This is only supported for user service managers (i.e. in conjunction with the --user option) or in containers and is equivalent to poweroff otherwise.
*
