We have pool of test targets. When we start a test job, then a vacant test target is looked up, and acquired.
Now, we could have created a gitlab runner for each target, and tagged them. But for two reason I would like to avoid that.
- A test may some times require more than one test target, then I would need to runners?
- A test target may be used with other systems or manual testing. In these cases the runner would need to be paused
So what we do right now, is to start the job with acquiring the resources. Now the problem is, that we need to wait for the resources. This wait time however is included in the allowed run time. So If the job wait for 50 minutes, it will only have 10 minutes left.
Increasing the allowed runtime to two hours however bites back, because a runaway job started 10 minutes before, then will keep the resource for two hours, and the waiting job will still only will have 10 minutes runtime left.
Besides that, the wait time, also ruins the execution time statistics of CI.
What could our solutions be.
- Could the runner wait for the resource, before allowing the job to run?
- Is it possible to reset the timer from within the job?
- Could the acquire be done in a before script, which is allowed to retry, but the main script is not allowed to retry?
Or somebody got a better idea?
Any good idea’s would be appreciated.