Custom Executor into Windows VM (Using Linux KVM/libvirt)

Hey, I have a Linux machine that I’ve got a couple VMs on. I’m using the libvirt example given in:

and it works great for my Linux VMs. I’ve also set up a Windows Server 2022 VM and installed on my dependencies on it. I’ve set up key-based SSH login to this machine that also works great. The difference seems to be that when I SSH into the Windows machine I get the cmd shell. So I’ve updated my runners config to be:

  name = "win2022-vm"
  url = "https://gitlab"
  token = "xxxx"
  executor = "custom"
  shell = "cmd"
  builds_dir = "c:\\builds"
  cache_dir = "c:\\cache"
  [runners.custom_build_dir]
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]
    [runners.cache.azure]
  [runners.custom]
    prepare_exec = "/vms/scripts_win2022/prepare.sh"
    run_exec = "vms/scripts_win2022/run.sh"
    cleanup_exec = "vms/scripts_win2022/cleanup.sh"

It still seems to be SSHing into the Windows machine with the wrong shell:

  on win2022-vm epb-hs2X
Resolving secrets
00:00
Preparing the "custom" executor
Using Custom executor...
Formatting '/vms/runner-1232-project-2316-concurrent-0-job-919213.qcow2', fmt=qcow2 size=214748364800 backing_file='/vms/win2022.qcow2' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off 
Starting install...
Domain creation completed.
Waiting for VM to get IP
VM got IP: 192.168.122.98
Waiting for sshd to be available
Preparing environment
00:00
The system cannot find the path specified.
ERROR: Job failed: prepare environment: exit status 1. Check https://docs.gitlab.com/runner/shells/index.html#shell-profile-loading for more information

Any ideas? Thanks!

I discovered that I was still using the /bin/bash command to the SSH call in the run.sh script. Oops. I changed it to cmd.exe and had it freeze up while trying to call git clone. I decided to bite the bullet and go to PowerShell 7 where it now says:

Preparing environment
PowerShell 7.2.5
Copyright (c) Microsoft Corporation.
https://aka.ms/powershell
Type 'help' to get help.
PS C:\Users\Administrator> echo "Running on $([Environment]::MachineName)..."
echo: The term 'echo' is not recognized as a name of a cmdlet, function, script file, or executable program.
Check the spelling of the name, or if a path was included, verify that the path is correct and try again.
PS C:\Users\Administrator> 

If I SSH into the machine and type echo I get it aliased to Write-Output as expected…

PowerShell 7.2.5
Copyright (c) Microsoft Corporation.

https://aka.ms/powershell
Type 'help' to get help.

PS C:\Users\Administrator> echo

cmdlet Write-Output at command pipeline position 1
Supply values for the following parameters:
InputObject: 

Output when SSHing directly:

PS C:\Users\Administrator> echo "Running on $([Environment]::MachineName)..."     
Running on WIN-KD6KPVQHACJ...

run.sh on host:

#!/usr/bin/env bash

currentDir="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
source ${currentDir}/base.sh # Get variables from base script.

VM_IP=$(_get_vm_ip)

ssh -i /root/.ssh/id_rsa -o StrictHostKeyChecking=no Administrator@"$VM_IP" pwsh.exe < "${1}"
if [ $? -ne 0 ]; then
    # Exit using the variable, to make the build as failure in GitLab
    # CI.
    exit "$BUILD_FAILURE_EXIT_CODE"
fi

I’m basically talking to the rubber ducky :stuck_out_tongue: I switched the shell in the config.toml to pwsh to match the fact that I’m using PowerShell 7. Now it freezes at the same spot as the cmd shell, trying to create the certifcate in c:\builds\tmp:

>> [System.IO.File]::WriteAllText("$CurrentDirectory/c:/builds/group/repo.tmp/CI_SERVER_TLS_CA_FILE", "-----BEGIN CERTIFICATE-----`nMIIGwTCCBamgAwIBAgIQIr2XWQH/Ti2wTUU4auG......

It looks like the script that is passed to pwsh.exe in the SSH command is cut short…is there such a thing as it being too long? It’s cut off part way through dumping the certifcate:

ssh -i /root/.ssh/id_rsa -o StrictHostKeyChecking=no Administrator@192.168.122.66 pwsh.exe
PowerShell 7.2.5
Copyright (c) Microsoft Corporation.
https://aka.ms/powershell
Type 'help' to get help.
PS C:\Users\Administrator> #!/usr/bin/env pwsh
PS C:\Users\Administrator> & {
>> 
>> $ErrorActionPreference = "Stop"
>> $FF_CMD_DISABLE_DELAYED_ERROR_LEVEL_EXPANSION="false"
>> $env:FF_CMD_DISABLE_DELAYED_ERROR_LEVEL_EXPANSION=$FF_CMD_DISABLE_DELAYED_ERROR_LEVEL_EXPANSION
>> $FF_NETWORK_PER_BUILD="false"
>> $env:FF_NETWORK_PER_BUILD=$FF_NETWORK_PER_BUILD...
more until we get to dumping the cert

then gets cut off. Since it’s not even sending the full Command-Let (or whatever pwsh speak is for the script) after & nothing gets executed and it hangs.

Definitely running into a character limit. I thought PowerShell didn’t have the same limit that cmd.exe did?

I seem to have the same problem, has this problem been solved?