We’re migrating from GitHub + CircleCI to GitLab CI (SaaS) and a few jobs in our build require more CPU cores and memory than others. In CircleCI you could specify the “resource_class” for each individual job. I can’t find anything similar in GitLab CI / Runners documentation.
Specific use case:
Our large Rails application has a lot of rspec tests. It’s best to run those tests in parallel and originally we were splitting them up by timings between different parallel jobs. This is not the most efficient way to do this because each job requires time to spin up and to load cache, so eventually we switched to double split:
- Split all test between 2 huge servers/runners (16 cores each).
- Within each job we took advantage of multi-threading to further split tests and run them in parallel.
The above improved out overall build times from 12 minutes to 4.5 minutes - without any increase in cost.
Possible solutions
One of the reasons we’re migrating to GitLab is the ability to bring our own runners/servers.
We’re planning to spin up a K8s cluster on DigitalOcean and “give” it to GitLab - to use exclusively to run CI jobs. K8s cluster will have finite resources: a few good size servers (8 core each).
Pod/Container specification in K8s allows user to specify the resources “request” and “limit”. Those could be used to spin up runner pods and let K8s figure out how many jobs a cluster can run in parallel - using the resource requests and limits.
Is this supported/possible?
What are the alternatives?
More questions
How does GitLab CI decide how many jobs it could run on a server?
How are jobs scheduled on K8s runners?
