If else in the script section of a job?

Good afternoon, was wondering if anyone has any tips on how to accomplish this? I have a job that I want to run a curl command on depending if a file called test.txt exists, like so:

test_job:
  stage: testing
  tags: Win-runner
  script: |
      if [ -f "test.txt" ]; then
          'curl https://www.website1.com';
      else
          'curl https://www.website2.com';
      fi

My builder fails with the following error:

$ if [ -f "test.txt" ]; then # collapsed multi-line command
(https://gitlab.com/myProject/-/jobs/12304#L26) bash:
line 76: curl https://www.website1.com : command not found

Any clues as I’ve been searching the docs and the web for a solution and nothing seems to stand out.

Thanks again.

The multi-line script approach looks good. I think the problem is with the curl command execution itself.

To rule out a dependency installation problem, I’d suggest to explicitly install curl before running any commands. I’m not sure which image or environment the job is running in, for Ubuntu/Debian based images it can look like this:

  before_script:
    - apt update && apt -y install curl 

Although, this isn’t the problem (I spun up a test playground project). The shell interpreter treats single quoted strings as “one execution command” - to make the error more visible, we can rewrite the commands with more arguments - it will still fail. Removing all arguments will work (curl found, but missing arguments error).

test_job:
  stage: test
  before_script:
    - apt update && apt -y install curl 
  script: |
      if [ -f "test.txt" ]; then
          'curl https://www.website1.com arg2 arg3 arg4';
      else
          'curl https://www.website2.com arg2 arg3 arg4 arg5';
      fi

image974×110 16.8 KB

Idea: Remove single quotes?

The shell interpreter should parse the command by itself, this will work. The argX passed will throw errors but they are different.

test_job:
  stage: test
  before_script:
    - apt update && apt -y install curl 
  script: |
      if [ -f "test.txt" ]; then
          curl https://www.website1.com arg2 arg3 arg4;
      else
          curl https://www.website2.com arg2 arg3 arg4 arg5;
      fi

Without single quotes, and solution

test_job:
  stage: test
  script: |
      if [ -f "test.txt" ]; then
          curl https://www.website1.com;
      else
          curl https://www.website2.com;
      fi

image745×186 23.7 KB

More ideas

Use double quotes for variable expansion, when necessary, i.e. “curl $HTTP_ADDRESS” if you want to populate the tested value from a global variable. It should not be necessary though.

I love challenges, so I built a matrix build version of the website tests too. Maybe that’s useful for your workflows too.

matrix_test_job:
  stage: test
  parallel:
    matrix:
      - DEVSECOPS_ADDRESS: [ https://about.gitlab.com, https://opsindev.news, https://o11y.love, https://qconlondon.com ]
  script: |
      if [ ! -f "test.txt" ]; then
          curl $DEVSECOPS_ADDRESS;
      fi

image742×380 28 KB

Technical background

If you are interested in learning why the shell interpreter fails with single quotes around the full command - for context, I was an OSS developer in my past job, and had to learn command parsing in C/C++ and ran into exactly these problems myself.

The underlaying technical problem is with how Linux shell execution works - the command strings are split by whitespaces, and the first argument in the list (argv[0] if you see that somewhere) being the binary to lookup, while all other commands (argv[1-n]) serve as additional command arguments. On the inside, the command parser converts this into a list of strings. The command in the solution could also be written to something like this:

'curl' 'https://www.website2.com' 

If you take away the work from the shell interpreter by “wrapping” the command into single quotes already,

'curl https://www.website2.com'

the shell interpreter thinks - hey cool, the first argument (argv[0]) is already defined, no split by whitespace necessary. Passes over the full string to execvpe() or popen() function calls, and this is what actually fails then, causing the terminal output error message.

Unfortunately, the command parsing is not “handed” to the final command string being executed. The function that throws the error does not know how the string was passed into the terminal from the outside. Which makes these types of errors hard to read and debug. Because really, the error message should read as

line 76: 'curl https://www.website1.com' : command not found. The command seems to be wrapped in single quotes, is this intentional? Could be a potential execution problem to troubleshoot. 

As developer, I’d try patching this into error messages. For curl, with many use cases, it probably is impossible to add. Different story though

Hope this little deep dive helps

Wow, thank you for the detailed answer. You really know your stuff, msg me if you run a class nearby I’ll try your suggestions and see which one works the best.

Thanks for your kind feedback. I love sharing my knowledge and experience in public, so that others can benefit and learn too.

If you want to meet in person - I’m usually around the Nuremberg area, Germany, and other locations when speaking at events. I will be at QCon London in 3 weeks, and KubeCon EU in Amsterdam mid April for example.

Need to go back to Europe to visit my family this year, will keep those events in mind. Thank you

The colon ( : ) character is special to the GitLab yaml file. Single quote the whole line and it should work. See the GitLab manual for this carefully hidden gotcha’.