Fail CI/CD jobs that remain pending for too long

I am using with a self-hosted Debian runner. I use this runner to run a backup job for a server on a private network.

I would like for this job to fail 6 hours after its queue time, regardless of whether it is still pending at that time, as it is crucial that these back-ups are being performed and that I am notified if they are not. However, so far the last queued job has been pending for over 19 hours without failing.

I’ve found a few issues (1, 2) related to having a separate timeout for queue time and execution time, which I think makes a lot of sense, however there does not appear t be a way to specify a timeout on the queue time of a job.

One possible way to get the desired behaviour would be to create a delayed job that simply checks on the backup job and fails if it has not completed after 6 hours, however this would be a little more complex and I was wondering if GitLab has a more elegant solution for this situation.

It looks like after about 24 hours or so the job was finally marked as failed, though my question still remains about how I might be able to control this timeout and cause it to happen sooner.


Thanks for the question! If these backup jobs are running often enough and not completing enabling auto cancelation of redundant pipelines could be a solution. This won’t satisfy your requirement of being notified if a previous pipeline failed so may not be a complete solution.

-James H, GitLab Product Manager, Verify:Pipeline Execution