Using case logic in an SSH connection with a here document

,

Case Logic inside Here Document running in SSH connection to server causes SSH scope to end

This may be more of a Bash question, but since I’m running this in a gitlab.com runner, I thought these fine folks might have an answer for me.

Background
I’m doing this as an open source demo of a gitlab.ci build and deployment on the IBM i platform, hosted on gitlab.com, with a connection to PUB400, a publicly available IBM i. Since IBM i can’t be used a runner, SSH is the best option that I have found.

I’m doing my best to present as secure of an SSH Connection as possible by not disabling secure host checking, cobbling together some examples I’ve found on the web, using an Alpine image for the gitlab.com hosted runner so that it will run lean and fast.

Requirements / Approach

  • All the runner really needs to do is connect to the IBM i over SSH where most of the heavy lifting will occur, then download any artifacts using SCP to save for the next deploy job, etc.
  • In my gitlab-ci.yml, since its for demo purposes, my intention is that it be self contained, so I’m not putting ci/cd logic in a bash script that would run on the SSH host, which could arguably make life easier.
  • All SSH secrets and Gitlab CI variables used in the .gitlab-ci.yml are described in comments at the top, since again its for demo purposes.
  • I’m using a Here Document to define the portion of the complex script that will run under SSH on the IBM i. I’ve found that this is the cleanest form for defining a complex bash script that runs over SSH.
  • I’m being careful to escape any variables in the Here Document that should not be expanded by the runner (in other words, variables that are used by the SSH host), while not escaping variables that should be expanded from Gitlab CI variables prior to the SSH. I’ve found this works well.
  • The here document needs to include some case logic that provides some flexibility for what happens on the IBM i host over SSH. This is where my problem is introduced.
  • I’m capturing the output from the SSH session in a bash variable so that take appropriate action in the runner. Basically, if an error occurs in the SSH host I want the runner to error, if the commands running over SSH are successful, I want the runner job to complete successfully. I do this by searching the output for specific error language, but I’m open to better ideas.

The Problem
I’m finding that the Here Document script runs fine on the IBM i host, to the point of the case logic’s end of scope. Any commands that should run after the fi are not run, and completely ignored.

What I am expecting
Any commands after the end of scope of the case logic running over SSH in a here document will still execute on the IBM i host.

Evidence
Here is a link to the pipeline failure.
Here is a link to the .gitlab-ci.yml used in the pipeline.

In the first link, you can see no output from the makei -v and makei b commands on lines 77-78:

When I move those commands before the fi on line 76, the pipeline is successful, as shown in this pipeline:

Why is the end of scope of the case logic also ending the scope of the here document and ssh session? This is making no sense to me.

Here is the full ci configuration for those that would rather not follow links:

Primary navigation
Homepage

# ######################################################################################################
# gitlab.com shared runners
# ######################################################################################################
# Variables you must set in Gitlab CiCd variables
#   IBMI_ADDRESS    - The DSN or IP address of the IBM i partition
#   IBMI_USERID     - The IBM i service account / user ID.  User have ability to SSH to your partition
#   IBMI_SSH_PORT   - The IBM i's SSH Port.  Usually '22', but PUB400.com has it as '2222'.
#   SSH_KNOWN_HOSTS - Run ssh-keyscan on your local machine to get the public key values from 
#                     the IBM i and store it as the SSH_KNOWN_HOSTS variable in Gitlab CICD settings.
#   SSH_PRIVATE_KEY - Run SSH-KEYGEN to generate a private / public key pair.
#                     The public key must be stored on the IBM i as an AUTHORIZED_KEY
#                     The private key must be stored as the SSH_PRIVATE_KEY variable in 
#                     Gitlab CICD settings
#   ############      #################################################################################
#   BUILD_DIR       - Required IFS Directory where the build will take place
#   OBJECT_LIB      - Optional static build library.  If not supplied, a temp library will be created
#
#                      
#######################################################################################################
image: alpine:latest # tell gitlab.com what linux image to run on.
stages:  # List of stages for jobs, and their order of execution
  - build
before_script: # Global script that runs before each job
  - apk update && apk add openssh-client bash tar git # install prerequisites
  
  - eval $(ssh-agent -s) # run ssh-agent
  # add base64 encoded private ssh key stored in SSH_PRIVATE_KEY variable to the agent store
  - bash -c 'ssh-add <(echo "$SSH_PRIVATE_KEY" | base64 -d)'
  - mkdir -p ~/.ssh # Hidden directory for ssh config
  # $SSH_KNOWN_HOSTS was output from 'ssh-keyscan -p 2222 pub400.com'
  - echo "$SSH_KNOWN_HOSTS" > ~/.ssh/known_hosts
build-job:
  # This job runs in the build stage, which runs first.
  stage: build
  script:
 #### Removed some commented out code from an earlier attempt at using a tarball instead of git.

    - echo ${BUILD_DIR}
    - echo ${CI_REPOSITORY_URL}
    - echo ${OBJECT_LIB}
    # ssh commands here   
    - echo "Compiling the code with SSH and Bob..."
    # Single line command to connect and run a series of commands over SSH
    - |      
      OUTPUT=$(ssh -p ${IBMI_SSH_PORT} ${IBMI_USERID}@${IBMI_ADDRESS} <<-ENDSSH1
        cd ${BUILD_DIR}
        export objlib=$OBJECT_LIB
        echo "objlib=\${objlib}"
        echo "PATH=\$PATH"
        rm -rf *
        git clone --branch ${CI_COMMIT_REF_NAME} ${CI_REPOSITORY_URL}
        cd ${CI_PROJECT_NAME}
        if [[ -z \${objlib} ]]; then
          export objlib=G#${CI_COMMIT_SHORT_SHA} 
          cl "CRTLIB \${objlib}" 
          echo "objlib \${objlib} has been created"
        else
          echo "clearing \$objlib"
          cl "CLRLIB \${objlib}"
        fi
        makei -v
        makei b
      ENDSSH1
      )
      # Check Output
      echo "${OUTPUT}"
      case "${OUTPUT}" in 
      *'successful'*)
        echo "Build Successful"
        ;;
      *)
        echo "Build Failed"
        exit 1
        ;;
      esac    
    - echo "Build job complete."

I have tried moving around the ‘)’ scope terminator for capturing the output to a variable, messing with indentation, among other things.

  • Thanks for any input :blush:*