Variables level on different stages

GitLab CI/CD
,
1

Hi,

I’m currently building a CI pipeline in order to automate tasks to build custom Debian ISO we used to do manually. I’m having some difficulties understanding variables “levels” in a job, I explain in brief my workflow:

  1. Job make-preseed-iso creates a custom preseed Debian ISO

  2. Job upload-preseed-iso upload the custom preseed ISO to Nutanix Prism Central through API call.

On job make-preseed-iso everything works fine right here. I only export the ISO_NAME in order to be used on the next job.

On the second job upload-preseed-iso , it retrieves artifacts from previous job, then I creates a json bash variable with JQ to correctly get the data format in order to be used by the curl command, which uploads the whole thing to Nutanix Prism Central and this is where I got some issues.

=> If I’m using variables on the jq cli like

  • --arg name "$ISO_NAME" => then, the variable can’t be retrieved.
  • If I do the same without variable --arg name "name-of-myISO.iso" => it works

Also, if I echo all variables I need in this job, they’re all displayed so variables seems to be pass from one to another job successfully.

Question :

How can I properly use variables inside a script and/or inside a command of a script ?

Also, how does variables are shown by stages and job if you use ““ , ‘‘ or no quotes ?

Thanks a lot !

Gael

default:
  image: debian:13.1

stages:
  - setup
  - build

variables:
  ISO_URL: "https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/debian-13.2.0-amd64-netinst.iso" 
  PRISM_CENTRAL: "https://XXXXXXX.com" 
  REMOTE_SFTP: yyyyy.fr 
  WORK_DIR: "/my_dir"
  SERVICE_ACCOUNT: "my_sa@mydomain.com"  
  SA_PASSWORD: 'XXXXXXyyyyy'
  DESCRIPTION: "Custom Debian"

# JOB TO CREATE THE CUSTOM ISO
make-preseed-iso:
  stage: setup
  before_script:
# [ .... ]

  script:
# [ .... ]
    - echo "ISO_NAME=$ISO_NAME" > iso.env  
  artifacts:
    reports:
      dotenv: iso.env  

# JOB TO UPLOAD THE CUSTOM ISO 
upload-preseed-iso:
  stage: setup
  needs:
    - job: make-preseed-iso 
      artifacts: true 
  before_script:
    - apt update && apt install curl jq -y
  script:
   # debug - echo "REMOTE_SFTP=$REMOTE_SFTP"
   # debug - echo "ISO_NAME=$ISO_NAME"
   # debug - echo "SERVICE_ACCOUNT=$SERVICE_ACCOUNT"
   # debug - echo "SA_PASSWORD=$SA_PASSWORD"
   # debug - echo "DESCRIPTION"=$DESCRIPTION"
    - | #build JSON data to correctly used the next curl command
      json=$(jq -n \
        --arg name "$ISO_NAME" \
        --arg url "http://$REMOTE_SFTP/os/$ISO_NAME" \
        --arg user "$SERVICE_ACCOUNT" \
        --arg pass '$SA_PASSWORD' \
        --arg description '$DESCRIPTION'
        '{
          name: $name,
          description: $description,
          type: "ISO_IMAGE",
          source: {
            "$objectType": "vmm.v4.content.UrlSource",
            url: $url,
            shouldAllowInsecureUrl: true,
            basicAuth: {
              username: $user,
              password: $pass
            }
          },
          clusterLocationExtIds: ["000AA63d-3444-35ce-0566-88e9a465d3fc"]
        }'
      )

    - | 
      curl --request POST \
      --insecure $PRISM_CENTRAL/api/vmm/v4.0.b1/content/images \
      --header 'accept: application/json' \
      --header 'authorization: Basic aaaaazzzzzeeeererererereerer' \
      --header 'content-type: application/json' \
      --header 'if-match: string_sample_data' \
      --header "ntnx-request-id: $(cat /proc/sys/kernel/random/uuid)" \
      --data "$json"
2

Id be tempted to throw in a - env | sort in the press job to verify the variables that have been passed.

3

I used echo for debugging and variables are passed, however within the jq command, it seems it doesn’t interprete them.

4

Hi there,

I did a little test of jq locally inside an alpine docker container: to me it seems like, when passing arguments:

  • –arg pass ‘$SA_PASSWORD’ → does not work
  • –arg pass “$SA_PASSWORD” → does work

So, my first suggestion would be to replace ' with " when passing arguments to jq

You can also echo your $json temporarily before sending it to curl, just to debug and see if everything got inside correctly. Once you got your json correct, then you can check if it’s passed into curl correctly.

2 Likes
5

thats a standard bash shell feature. I would have mentioned it if I saw single quotes in the sample.

e.g.

FOO=bar
echo "$FOO"
bar
echo '$FOO'
$FOO

For the most part GitLab ignores $VAR variable directives in ci script, so $VAR declarations in the script:section get passed to the shell executor to parse and thus its rules apply.

1 Like
6

Thanks for the updates guys ! Indeed it was a matter of quotes !

I finally build my payload.json with jq and the right quotes, then check it with jq and finally pass my file in curl argument and everything is fine :slight_smile:

1 Like