How to use the "cache" in gitlab when trying to use files from two previously run jobs?

GitLab CI/CD
1

I have the following .gitlab-ci.yml configuration:

stages:
  - stage1
  - stage2
  - stage3

job1:
  stage: stage1
  rules:
    - if: $RUN != "job3"
      when: always
    - when: never
  script:
    - date > data/test1.txt
  cache:
    key: test-cache
    paths:
      - data/test1.txt

job2:
  stage: stage2
  rules:
    - if: $RUN != "job3"
      when: always
    - when: never
  script:
    - date > test2.txt
  cache:
    key: test-cache
    paths:
      - test2.txt

job3:
  stage: stage3
  rules:
    - if: $RUN == "job3"
  script:
    - cat data/test1.txt
    - cat test2.txt
  cache:
    key: test-cache
    paths:
      - data/test1.txt
      - test2.txt

The plan is to do the following:

  1. To run the pipeline with the variable RUN not defined, so that ONLY the two jobs job1 and job2 are run, but NOT job3. The two jobs job1 and job2 are supposed to add the files data/test1.txt and test2.txt to the cache, so they can be fetched later by job3.
  2. To run the pipeline with the variable RUN set to run3, which ONLY runs the job job3. This job is supposed to fetch the previously generated files from the cache, namely the two files data/test1.txt and test2.txt.

Explanation:

In reality, the two jobs job1 and job2 are very time consuming jobs which also consume a lot of API calls. Therefore, I do not want to rerun them for no reason, as they would generate the same results anyway. I just want to “store” the results from these two jobs job1 and job2, and “fetch” these results from a later running job (maybe a few days later). This job3 will also run in the same pipeline in the same repo and on the same branch. as the jobs job1 and job2.

Expected result:

job3 can use the two files from the cache created by job1 and job2.

Observed result:

$ cat data/test1.txt
cat: data/test1.txt: No such file or directory

I have the following questions:

  1. How can I fix this problem? What IS the problem?
  2. Where is this documented?
2

Use artifacts for passing build artifacts from job1 to job2

3

Artifacts is not the correct tool to use. Please read the question again.

4

I think you are wrong. Artifacts is exactly that tool for passing result of stage1:job1 to stage2:job2. Cache is more suitable for storing dependencies that can be shared between different jobs and even different pipelines. Can you please explain why artifacts is not suitable for you, maybe i don’t understand all aspects of your problem.

5

I want to run some jobs of the pipeline which create some artifacts.

I want to use these artifacts in a different job which I run at a later time.

Maybe artifacts is the right tool, but as I understand gitlab (not sure a single person can understand this ever), this is not possible.

If you think this is possible I strongly appreciate a complete configuration I can try myself. Thank you

6

It looks like I overlooked some statement somewhere mentioned in the incredible vast gitlab documentation: It says

If two jobs have the same cache key but a different path, the caches can be overwritten.

The solution was to add the following syntax to each of the three jobs:

  cache:
    key: test-cache
    paths:
      - data/test1.txt
      - test2.txt

In that case, job3 was able to use the files that were created in the other two jobs.

Of course, that is no guarantee if it works the same way when I try to implement the exact same structure to the real pipeline…