Build job fails randomly with no changes to code

GitLab CI/CD
, ,
1

I have a scheduled pipeline that runs everynight at 10PM UTC on a specific branch, nightly-tests. This pipeline has 2 jobs: Build and Test.

Over the 6 days, there has been no changes to this branch. However, the Build job has failed on day 2, 3, 4, and 6. Judging from the build output, the job is using old code from ~2 weeks ago.

I believe that the Build job on the scheduled pipeline is using build cache from a old Build job that failed on a different pipeline, but I can’t prove this.

I even added cache: [] to the Build job inorder to force the job to not use cache but the jobs still failed randomly.

What could I be doing wrong? I am more than happy to provide more info and context if needed. I have also left the pipeline config file for the scheduled pipeline below. Thank you!

# This pipeline config is for scheduled tests. They run on their own scheduled pipeline and are separate from
# the C++ testing. This is required since gitlab can only schedule pipelines, and not specific jobs


default:
  image: registry.gitlab.com/my-company/my-repo/my-image:1.5.0
  # Cache build objects on each new branch to decrease build time
  cache:
    key: ${CI_COMMIT_REF_SLUG}
    paths:
      - build
  timeout: 10hr
  artifacts:
    expire_in: 1 day

stages:
  - Build
  - Tests

Build:
  stage: Build
  cache: [] # Force this job to not use cache
  script:
    - cmake -E make_directory build
    - cd build && cmake -DOUTPUT_PATH_PREFIX=$(pwd) ../ -DCMAKE_BUILD_TYPE=Debug
    - cmake --build . --config Debug -- -j$(nproc)
  artifacts:
    paths:
      - "build"

.Integration Tests:
  stage: Integration Tests
  image: selenium-simulator-testing:latest
  services:
    - name: docker:20.10.16-dind
      alias: dockerhost
    - name: selenium/standalone-chrome:latest
      alias: chromehost
  needs:
    - job: Build
      artifacts: true
  variables:
    DOCKER_HOST: tcp://dockerhost:2375/
    DOCKER_TLS_CERTDIR: ""
    REPO_DIR: /builds/my-company/my-repo
  only:
    - nightly-tests
  tags:
    - simulator-e2e
  artifacts:
    when: always
    paths:
      - "${REPO_DIR}/integration_tests/test.log"
    reports:
      junit: "${REPO_DIR}/integration_tests/report.xml"
    expire_in: 3 day
  before_script:
    # move ci build output to from build/output/bin/debug to output/bin/debug
    - mkdir -p "${REPO_DIR}/output/bin/debug"
    - cp "${REPO_DIR}"/build/output/bin/debug/* "${REPO_DIR}/output/bin/debug/"
    # authenticate client docker container
    - mkdir -p "${HOME}"/.docker/
    - echo "${DOCKER_AUTH_CONFIG}" > "${HOME}/.docker/config.json"


Tests:
  extends:
    - .Integration Tests
  timeout: 10hr
  script:
    - cd "${REPO_DIR}/integration_tests/"
    - pytest --tb=short -vvv --junit-xml=./report.xml --retries 2 tests/**/*.py