Regularly rerun CI tests

The world changes around us!

Some of our code depends on the outside world (some even on stuff that is still fluctuating).

As an example I’ve just spend a day finding (and a couple of minutes fixing) a bug in one of our chef cookbooks that suddenly broke because of a change in the development version of Ubuntu (that will become a LTS version, so something we will probably run quite a few copies of). And I hit that because I changed something else in that chef cookbook, but couldn’t get on with that because the CI tests started failing.

It probably wouldn’t have saved me any work/time, but it would have saved some frustration if I hadn’t hit, and had to fix, a bug that was completely unrelated to what I was working on now - and I could if I had become aware of that change a month ago (when it was introduced). I would have if GitLab regularly rerun CI tests on this project. Can I make GitLab do that?

Hi,

you can schedule CI pipeline runs in a defined interval, similar to a cron job. We do that e.g. to check whether we need to build daily snapshot packages for our nightly build testers. I’d say you can incorporate this logic to re-run a failed CI pipeline/job as well.

https://docs.gitlab.com/ee/user/project/pipelines/schedules.html

Cheers,
Michael

1 Like

Why didn’t I look there? Okay, nobody has a chance of answering that.

I would’ve preferred it if I could have made a schedule that ran the job if it had been more than e.g. 7 days (so changes to the project, that causes the pipeline to run would delay the next automated run, that would probably also help distributing the load by running the pipelines at different times - there should probably also be some logic to do automatic runs at night, where there are way less load on our GitLab server). But for now I’ve created a schedule for one project to run sunday morning - I’ll delay marking your answer as the solution until I’ve seen on monday that it actually worked.

Hi,

it definitely is an interesting aspect to bring some more logic to pipeline schedules, especially with the failed scenario and retry iterations.

I haven’t dug deep into the rules added with 12.3, so they may bring some additional ideas here.

In terms of performance of the runners, the auto-scaling options with assigned resources in offpeak times might be interesting. That’s of course only an option if you have the capabilities for the runners e.g. in AWS or OpenStack or any other cloud. I have written a blog post on OpenStack lately.

Let’s see how your tests go, and if more logic is needed, we may generate some ideas and content for a feature request for GitLab then :slight_smile:

Cheers,
Michael

Well the job ran (and failed - as could be expected given that the fix I made last week hadn’t been merged yet) this morning.

It seems only I got the report, which is not so good - in this case it was fine, but given that some coordination will be required to make sure all projects aren’t build at the same time, I was hoping that I would be able to set it up for every project (also as a potential benefit for moving the projects to the new GitLab server I set up for this kind for project), but I don’t want reports for every project (and those in charge of each project will probably also prefer to be notified directly), it would also be preferable if each report were sent to multiple people (so someone would know of problems if the owner of that pipeline is e.g. on vacation - we have mailing lists/aliases so GitLab would only need to associate one address with each automatic pipeline).

The rules didn’t seem like anything that could help me, and things need to stay on our servers, so no cloud solutions for scaling here.

A feature request will be needed - i.e. if noone points to an already existing, that they might just have made, I’ll make one soon.

I should probably link to the issue I created:

1 Like