Hi
I have a custom docker image (simplified):
FROM ubuntu:19.10
# Update anything that might be out of date
RUN apt-get -qq update \
&& apt-get clean
# Compiler agnostic requirements
RUN apt-get install -y --no-install-recommends \
git \
python3 \
python3-pip \
&& apt-get clean
# Update pip to avoid error messages
RUN pip3 install --no-cache-dir --upgrade \
pip
ENTRYPOINT ["/bin/bash", "-c"]
CMD ["/bin/bash"]
If I use this image on the shared runners it fails with
sh: 3: set: Illegal option -o pipefail
The script is running with /bin/sh, which links to /bin/dash, which doesn’t have the pipefail option.
Strangely enough, if the ENTRYPOINT and CMD lines are removed from the docker file, it then runs successfully. Either by not trying to set pipefail during the gitlab generated script or by using bash.
I can continue working without the entrypoint, but I am curious as to why this is happening.
Thanks in advance
Paul
2 Likes
It sounds a bit strange indeed. I can reproduce it within our configuration. /bin/bash is the default shell for gitlab-runners to use. So I should expect it to use bash and not fail on a pipefail command.
But note that it only make sense specifying an ENTRYPOINT if you want to run a specific script. CMD ["/bin/bash"] is already specified in ubuntu:19.04 so you don’t need it. If you specify “ENTRYPOINT”, it basically means that the script you are running is already inside the image and specified in your ENTRYPOINT. See also: https://docs.gitlab.com/runner/executors/docker.html#the-entrypoint. I don’t know exactly how they implemented it, but in any case, it won’t run the script you specified in the .gitlab-ci.yml.
See https://gitlab.com/gitlab-org/gitlab-runner/issues/1170 for a longer discussion.
It seems like you’ve already found the solution? 
Yes. I know the solution, but I am was wondering why this was happening.
The linked issue seems to have been closed prematurely as the issue persists. Not a serious bug, but the workaround is not obvious for someone that encountered the error the first time.
1 Like
Since this has been annoying me for approximately a week and this is the only remotely relevant page that comes up when searching for “gitlab-runner illegal option” or “gitlab-runner pipefail”, I figured it’d be worth reviving this thread to help increase visibility since for the past ~5yrs that this issue’s been known GitLab seems to show no intent to fix or even document this issue.
Here’s a list of issues sharing the same root cause as this one, even though the symptoms being reported differ:
Another fed-up user and myself have even submitted an MR recently that should address this issue, though we may be missing information as we have our own jobs to do: update bash.go: remove bashisms (!3014) · Merge requests · GitLab.org / gitlab-runner · GitLab
For those wanting to save a click, the existing workaround is to leave ENTRYPOINT and CMD undefined in your Dockerfile when making a custom image or override the ENTRYPOINT value with the entrypoint: [""] directive available in the image: section of .gitlab-ci.yml. This does NOT fix the logic to spawn an appropriate shell, it just appears to allow the runner to proceed with bash(1), which is hopefully acceptable in your deployments.
As for why specifically the error is: “Illegal option pipefail”, it’s because the runner injects shell scripts prior to running anything specified in the script: section(s) of .gitlab-ci.yml and will make a call like so: sh -c set -eo pipefail, assuming that sh(1) is a symlink to bash(1) or supports the nonstandard option -o pipefail as with Busybox ash(1), but NOT dash(1) as is the default on Debian and Debian-derived systems.
If you find this thread and the above mentioned issues still aren’t fixed, I can only suggest that you not give GitLab your money as there’s no excuse for there to have not been as much as a documentation update warning users about exactly this issue opened 5yrs ago with several related issues popping up. I’m not familiar with other CI offerings but I’d recommend at least giving them a second look so long as these issues remain open and UNDOCUMENTED. If you already have a contract with GitLab, you may want to reach out and ensure they’re working on at the very least fixing their documentation to mention that they only support shells with -o pipefail.
1 Like
I hate to necropost, but I’d argue this is in fact a serious bug as it’s both hard to search for and all existing documentation is simply wrong, as we don’t have the option to specify shell = "sh" in the config.toml, the gitlab-runner process still generates set -eo pipefail after explicitly executing sh(1).
This behaviour isn’t documented, the troubleshooting doesn’t mention it, and the shell documentation implies that the shell choice can be overridden, but it still assumes it’s running as bash(1).
Less serious than something that renders the gitlab-runner unusable for sure, but still a serious mismatch between documentation, stated support, and observed behavior.
1 Like