Gitlab Runner virtualisation approaches

Hi all,

We currently have over 350 Gitlab Runners hosted on individual instances in EC2. Our jobs require a ~38 Docker container Compose environment to run in the most part. We use these instances as a single Gitlab Runner shell executor.

We’re considering bringing these runners back in-house, setting up our own mega-servers with over 1,000 CPU threads and 1TB RAM.

What options do we have to spin up runners capable of replicating this Compose environment?

In summary, we need ~38 containers to spin up for every single job in our pipeline. Given 700+ jobs and ~5 concurrent pipelines (~15 successful per day), this is quite a hefty and complex task. These must be a fresh spin-up of the environment too as some of our tests change our data storage (MySQL, Redis, Postgres etc).

I can think of:

  • Docker in Docker approach (probably using something like Kaniko)
  • Container orchestration (especially Kubernetes for the kubernetes executor)
  • Virtualisation at the operating system layer, e.g. VMWare, Proxmox etc

Multiple Gitlab Runners on one box wonldn’t work on its own as they wouldn’t be able to run multiple simultaneous Compose environments.

I’d be particularly interested in hearing from any Linux sysadmins out there about the OS-level virtualisation approach. For this I think we’d need something with an API - or at least the ability to script the creation of so many virtual servers.

Thanks in advance,
Duncan