*Hey, I have a need to use a cache mechanism between different jobs for specific ETL processes. I have configured my own runners with a shared cache based on S3. I’m using Gitlab.com.
One of my main jobs (dbt_test) starts as scheduled on the main branch and puts objects in the cache under the “main” key, for example. The file on the S3 bucket is named: “main-protected”.
At moment when I want to refer to this cache key on another branch (e.g., during MR - job called dbt_dry_run), unfortunately, I get an error because typing the value: main in the key attribute, Gitlab looks for the file: main-unprotected which is not my intention.
Is it possible to somehow change this behavior so that it can refer to the value of the key in the cache created by the main branch?
Parf of my .gitlab-ci.yml:
dbt_test:
...
stage: stg_test
rules:
- if: $CI_PIPELINE_SOURCE == "schedule" && $stg == "true"
allow_failure: true
tags:
- dbt
cache:
key: "$CI_COMMIT_REF_SLUG-stg"
paths:
- outputs/
policy: push
when: always
script:
- cp -a target/ outputs/
- dbt test
dbt_dry_run:
...
stage: mr
rules:
- if: $CI_PIPELINE_SOURCE == "merge_request_event" && $CI_MERGE_REQUEST_TARGET_BRANCH_NAME == "main" && $CI_MERGE_REQUEST_LABELS == "test"
cache:
key: "main-stg"
paths:
- outputs/
policy: pull
when: always
tags:
- dbt
script:
- ls outputs/