Reusing artifacts in pages

GitLab CI/CD
1

I’m trying to build documentation artifacts and deploy it to a preview app in one job, and then deploy those same artifacts to pages (so later I’ll only build pages for the main branch). However, I cannot for the life of me figure out why the pages job can’t access the artifacts created during the previous stages.

Here’s a workable simplified version of my .gitabl-ci.yml file:

stages:          # List of stages for jobs, and their order of execution
  - build
  - deploy

dbt-docs-preview:
  rules:
    - if: $CI_MERGE_REQUEST_ID

  stage: build
  image: python:3.7

  variables:
    DBT_PROFILES_DIR: "${CI_PROJECT_DIR}/dbt"

  environment:
    name: preview/${CI_PROJECT_NAME}/${CI_COMMIT_REF_SLUG}/dbt-docs
    url: "https://$CI_PROJECT_NAMESPACE.gitlab.io/-/$CI_PROJECT_NAME/-/jobs/$CI_JOB_ID/artifacts/public/index.html"
    on_stop: dbt-docs-stop

  script:
    - pip install dbt-bigquery
    - cd dbt
    - dbt deps --target=ci
    - dbt docs generate --target=ci
    - mkdir -p ${CI_PROJECT_DIR}/public
    - cd target
    - cp index.html catalog.json manifest.json run_results.json ${CI_PROJECT_DIR}/public

  artifacts:
    paths:
      - public

dbt-docs-stop:
  stage: build
  image: python:3.7

  rules:
    - if: $CI_MERGE_REQUEST_ID
      when: manual

  allow_failure: true

  environment:
    name: preview/${CI_PROJECT_NAME}/${CI_COMMIT_REF_SLUG}/dbt-docs
    action: stop

  script:
    - echo "Removing dbt-docs preview"

dbt-docs-deploy:
  rules:
    - if: $CI_MERGE_REQUEST_ID

  stage: deploy
  image: python:3.7

  environment:
    name: production/${CI_PROJECT_NAME}/dbt-docs
    url: "https://${CI_PROJECT_NAMESPACE}.gitlab.io/-/${CI_PROJECT_NAME}/-/jobs/${CI_JOB_ID}/artifacts/public/index.html"

  script:
    - echo "Deploying dbt-docs"

  artifacts:
    paths:
      - public


pages:
  stage: deploy

  needs:
    - dbt-docs-preview

  script:
    - echo "I should be using artifacts from previous jobs"

  artifacts:
    paths:
      - public

When I use gitlabs CI editor, it says the yaml is valid and shows the proper dependencies:

ci_vizualize
ci_vizualize978×588 30.6 KB

However, when I commit the change, two pipelines are created, and one of them immediately fails due to a yaml parsing error:

multiple_pipelines
multiple_pipelines1996×410 66.6 KB

error
error1272×712 56.2 KB

Of course, the needs keyword shouldn’t be needed here because the jobs are in separate stages. However, if I remove it then the pages job completes before the documentation build is complete because it’s in a separate pipeline, so of course it has no artifacts to publish to pages.

Could there be something wrong with my configuration that is forcing these jobs into separate pipelines. I’ve found other examples (1, 2) of people deploying this strategy, but for some reason it’s not working for me.

2

:exploding_head: OMG, I went down a dozen rabbit holes today trying to get this working. 5 minutes after writing this post, I stumbled upon the solution: rules!!! The pages section didn’t have any rules defined so it was running as a separate pipeline for some reason. If I add the same rules to pages, then it all runes in the same pipeline. I was holding off on writing the rules because I knew they would end up being different and I just wanted to get the basic mechanics working.