CI/CD Debugging scheduled pipelines

GitLab CI/CD
1

Unpredictable behavior with scheduled jobs in self-managed instance of gitlab

Hello,
I have observed unpredictable behavior from scheduled jobs in gitlab. The Yaml syntax is correct, and the job manually runs (by pushing the play button). Also push pipelines work correctly (when we push to the repo), and sometimes the scheduled pipelines work, but now I am stuck in a state where I cannot make them run and I dont know what to do.

The strange point is that the deadline of the scheduled pipeline passes without going into the execution queue, and the time of the “Next Run” wont get updated to the next period.

Example:

I create a daily job to run every night at 9:13PM (lets assume that now it is 9:10PM).
So in 3 minutes the job should start. But the pipeline doesnt start and the “Next Run” remains the same day without being rescheduled for tomorrow.

image
image1544×107 7.91 KB

I have tried to simplify the pipeline and removed all rules. Still the pipeline doesnt run, while a more complex pipeline has been running before that without any issues.
Also I have made sure that the branch is not protected.

Do you have any recommendations on how to debug this issue? Are there any debug commands / tools to understand what is going on.

Here are the versions that we use:

  • *GitLab (Hint: /help): GitLab Community Edition [14.3.6]
  • *Runner (Hint: /admin/runners): 14.1.0

And here is the .gitlab-ci.yml that I use for this experiment:

image: "..."

variables:
    GIT_UPDATE_SUBMODULES: "git submodule update --init --recursive"
    GIT_STRATEGY: clone

stages:
    - setup
    - deploy

default:
    before_script:
        # Force automount on the following directories
        - ls -l /local/user/

setup:env:
    stage: setup
    tags:
        # Note: tags can be space separated. Each line indicates a single tag!
        - icdesign
    before_script: []
    retry:
        max: 2
        when: always
    script:
        - echo "PYTHON_EXEC=python3" > setup.env
    artifacts:
        reports:
            dotenv: setup.env

setup:profile_ci:
    stage: setup
    tags:
        - icdesign
    before_script: []
    retry:
        max: 2
        when: always
    script:
        tcsh -xvtc "         &&
        $GIT_UPDATE_SUBMODULES &&
        source digital_flows/directives/ci/create_profile_ci.csh"
    artifacts:
        paths:
            - profile_ci.csh
        expire_in: 1 day

job1_rtl_questa:
    stage: deploy
    tags:
        - icdesign
    needs: ["setup:env", "setup:profile_ci"]
    script:
        tcsh -xvtc "         &&
        $GIT_UPDATE_SUBMODULES &&
        source digital_flows/directives/ci/prepare_target.csh &&
        tcsh -b directives/cron/job1_full_regress.sh"
    artifacts:
        paths:
            - cron_log.txt

        expire_in: 5 day
  • What troubleshooting steps have you already taken? Can you link to any docs or other resources so we know where you have been?

Thanks for taking the time to be thorough in your request, it really helps! :blush: