CI/CD doesn't want to use bash, how to fix

Describe your question in as much detail as possible:

I have quite simple CI/CD job definition, here it is

image:
  name: aquasec/trivy
  entrypoint: [""]
sast-scan-trivy:
  variables:
    GIT_STRATEGY: fetch
    GIT_CHECKOUT: "true"
  before_script: 
    - apk update
    - apk add git bash ca-certificates
    - cp /etc/gitlab-runner/certs/ca.crt /usr/local/share/ca-certificates
    - update-ca-certificates
  script:
    - >
      #!/usr/bin/env bash

      export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
      export COMMIT_BEFORE_SHA="$(git rev-parse HEAD~1)"
      export COMMIT_SHA="$(git rev-parse HEAD~0)"
      export COMMIT_BEFORE_SHA="$(git rev-parse HEAD~1)"
      export COMMIT_SHA="$(git rev-parse HEAD~0)"

      FILES=`git diff "${COMMIT_BEFORE_SHA}" "${COMMIT_SHA}" --name-only`

      for FILE in "${FILES[@]}"; do
          # Check if the file exists
          if [ -e "$FILE" ]; then
              echo "Scanning file $FILE"
              trivy fs --scanners vuln,secret,misconfig "$FILE"
          else
              echo "File $FILE does not exist."
          fi
      done
  • What are you seeing, and how does that differ from what you expect to see?

I tried the same job with the latest Ubuntu image, and it just works.
In Apline it simply refuse to work.

  • Consider including screenshots, error messages, and/or other helpful visuals

The job is always failing with:

$ cp /etc/gitlab-runner/certs/ca.crt /usr/local/share/ca-certificates
$ update-ca-certificates
$ #!/usr/bin/env bash # collapsed multi-line command
/bin/sh: eval: line 156: syntax error: bad substitution
Cleaning up project directory and file based variables 00:01
ERROR: Job failed: exit code 2
  • What version are you on? Are you using self-managed or GitLab.com?

Self-managed version of Gitlab, the latest one.

  • Runner (Hint: /admin/runners):

I have Docker executor runner, check its configuration below:

concurrent = 1
check_interval = 0
shutdown_timeout = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "sofx1013dckr309.home.lan"
  url = "https://gitlab.home.lan"
  id = 12
  token = "glrt-yNzEHJxP9RnddS2jx19H"
  token_obtained_at = 2024-01-28T17:17:50Z
  token_expires_at = 0001-01-01T00:00:00Z
  tls-ca-file = "/etc/gitlab-runner/certs/ca.crt"
  executor = "docker"
  [runners.cache]
    MaxUploadedArchiveSize = 0
  [runners.docker]
    tls_verify = false
    image = "ubuntu:latest"
    privileged = false
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/cache", "/etc/gitlab-runner/certs/ca.crt:/etc/gitlab-runner/certs/ca.crt:ro"]
    shm_size = 0

Please help with this.
Thanks.

Hi,

I’m not 100% sure, but I believe you have run into a typical shell vs bash script issues.

Please note: /bin/sh and /bin/bash are not the same.

Ubuntu comes with both, and defaults to /bin/bash, which has more functionality then /bin/sh.

Alpine comes only with /bin/sh and therefore it’s a bit limited. I am master of neither, so I have no clue what is wrong out of the box.

But, I’d start by replicating this locally (docker run -it -v ${pwd}:/src alpine sh) and typing in command by command to see where it fails and why.

Good luck!

Anyway,
I have managed to fix it by using the code below:

      FILE_LIST=`git diff "${COMMIT_BEFORE_SHA}" "${COMMIT_SHA}" --name-only`

      for FILE in $FILE_LIST; do
          # Check if the file exists
          if [ -e "$FILE" ]; then
              echo "Scanning file $FILE"
              trivy fs --scanners vuln,secret,misconfig "$FILE"
          else
              echo "$FILE does not exist."
          fi
      done

You are right about sh and bash difference, but I expect when I have !/bin/bash at the top of the script that script to be executed by using bash shell.

The first thing I noted was that before_script and script are indented below sast-scan-trivy. It seems it works anyway, so maybe it’s just the copy-paste to here that’s gone wrong? (or maybe it’s a thing I just find weird because I don’t use SAST.)

But you don’t, you have #!/usr/bin/env bash - which is about the same, but I would still try changing it.

Another thing that makes me wonder: The error message talks about line 156, there is not 156 lines in the script you show.

Yeah I’m also curious about the issue in line 156, where there is no such line.

Yesterday I have checked these variables in order to get some clue about the issue:

    FF_PRINT_POD_EVENTS: "true"
    FF_USE_POWERSHELL_PATH_RESOLVER: "true"
    FF_SCRIPT_SECTIONS: "true"

Unfortunately they don’t make any difference in regards of the output. The Runner is executing the scripts and returns only the output on the web console, instead of all the commands executed.

I even tried to use bash -x, to make bash more verbose, but no difference at all.

So at this point troubleshooting Gitlab pipelines are a big black-box for me :wink:

I dug around this a few weeks ago because I also found myself limited by the Almquist shell (for all intents and purposes, basically the same as the Bourne shell or sh) in the same way as you (i.e. lack of support for arrays and associative arrays). Along the way, I got curious about how exactly the script given to the job is executed by the runners. I went into the source code of various container based Jobs executor, and ran multiple times in something like the following just before script execution:


Basically, the executors try to use bash if it is installed in the image to run the given job script, falling back to sh if that is not available. Shebang doesn’t really work there: we are not executing the script directly as if it was an executable file, we are passing it to the sh or bash binary. Even worse, before_script is appended to script before you start running it, so that Shebang might even be in the middle of the script depending on your job configuration. Long story short: if you want to use Bash, you need to use an image with bash on it BEFORE the job starts. Or you can do some redirection magic and call bash explicitly, but I wouldn’t really recommend that.

1 Like

It seems that the problem is with the Alpine container image from trivy upstream, and limited sh support. Maybe an alternative can be to create a Debian-slim based image and install trivy from packages, and use that as a CI/CD job image.

Looking at how the alpine trivy images is built, it looks relatively straight-forward because it only copies the trivy Go binary, and some additional assets. trivy/Dockerfile at main · aquasecurity/trivy · GitHub