Gitlab CI - Custom VM runner, somehow like SSH, but with custom "prepare" script

GitLab CI/CD
,
1

I am trying to setup a VM-based set of runners with clean environment on for job. Just like virtualbox runner, but I have two dedicated self-hosted machines both running proxmox which uses KVM.

I have multiple VMs setup; a manager VM (the one that the runners are registered on) and multiple other VMs for each platform, each one updated and setup with some basic packages installed and then snapshot.

VMs screenshot and runner configs

image

Config script: https://hastebin.app/5e7246363f5e83001c1b9692
Prepare script: https://hastebin.app/5e7245933f5e83001c1b968f
Cleanup script: https://hastebin.app/raw/5e7245c23f5e83001c1b9690
Sample runner config.toml: https://hastebin.app/5e7246253f5e83001c1b9691

Current VS expected

  • VM runner0.gitlab.domain.tld is the runner manager VM
  • bold text is the one that is different

Current CI flow
  1. pipeline is triggered
  2. job is assigned to a runnerX on VM runner0.gitlab.domain.tld
  3. script “config.sh” is run on runner0 (sets the variables)
  4. script prepare.sh is run on runner0
    1. ssh to vm host box
    2. force-stop runnerX vm
    3. rollback runnerX vm to snapshot “mainRunner”
    4. startup runnerX vm
    5. wait until able to SSH to runnerX
  5. gitlab-runner on runner0 downloads artifacts, cache, git codebase
  6. gitlab-runner on runner0 runs the .gitlab-ci.yml scripts
  7. script “cleanup.sh” is run on runner0
Needed CI flow
  1. pipeline is triggered
  2. job is assigned to a runnerX on VM runner0.gitlab.domain.tld
  3. script “config.sh” is run on runnerX (sets the variables)
  4. script prepare.sh is run on runner0
    1. ssh to vm host box
    2. force-stop runnerX vm
    3. rollback runnerX vm to snapshot “mainRunner”
    4. startup runnerX vm
    5. wait until able to SSH to runnerX
  5. gitlab-runner on runnerX downloads artifacts, cache, git codebase
  6. gitlab-runner on runnerX runs the .gitlab-ci.yml scripts
  7. script “cleanup.sh” is run on runner0

What I need is somewhat like the SSH runner, but with “prepare script” added. I don’t even need the config and cleanup script. I would only need the prepare one.

Extra details:

  • Using KVM/qemu for virtualization (proxmox to be mroe specific)
  • I am using selfhosted gitlab-ce (12.8.6 5fc76a64537)

Thank you in advance for any reply and your time for reading this.

4

After a few days, I managed to set everything up and wrote an article about it.
You can read it here; https://aljax.us/how-to-setup-gitlab-runners-in-kvm-qemu-virtual-machines/

1 Like