Fail CI/CD jobs that remain pending for too long

I am using GitLab.com with a self-hosted Debian runner. I use this runner to run a backup job for a server on a private network.

I would like for this job to fail 6 hours after its queue time, regardless of whether it is still pending at that time, as it is crucial that these back-ups are being performed and that I am notified if they are not. However, so far the last queued job has been pending for over 19 hours without failing.

I’ve found a few issues (1, 2) related to having a separate timeout for queue time and execution time, which I think makes a lot of sense, however there does not appear t be a way to specify a timeout on the queue time of a job.

One possible way to get the desired behaviour would be to create a delayed job that simply checks on the backup job and fails if it has not completed after 6 hours, however this would be a little more complex and I was wondering if GitLab has a more elegant solution for this situation.

It looks like after about 24 hours or so the job was finally marked as failed, though my question still remains about how I might be able to control this timeout and cause it to happen sooner.

@kyle_anderson

Thanks for the question! If these backup jobs are running often enough and not completing enabling auto cancelation of redundant pipelines could be a solution. This won’t satisfy your requirement of being notified if a previous pipeline failed so may not be a complete solution.

-James H, GitLab Product Manager, Verify:Pipeline Execution