Git clone in VirtualBox hangs
I’ve been trying to switch from docker-windows executor to virtualbox executor
At the moment the gitlab runner is able to instantiate the virtualbox clone and communicate with it through SSH by forwarding the port 22 to another port as the documentation request.
My problem comes when I try to execute SSH within the Job Script found in .gitlab-ci.yml.
Our project has some dependencies that requires to be obtainer from other our own repositories… we use Unity and our only option at the moment is to rely on SSH so HTTPs is not an option.
Here is the .gitlab-ci.yml I’m using to debug the issu
workflow:
rules:
- if: $CI_COMMIT_BRANCH == "main"
- if: $CI_COMMIT_BRANCH == $ALLOWED_BRANCH
- if: $ALLOW_BRANCHES == "true"
stages:
- build
build: # This job runs in the build stage, which runs first.
stage: build
image:
name: Win10-22H2-Unity
pull_policy: always
before_script:
- cp $env:SSH_KEY $env:USERPROFILE/.ssh/id_ed25519
# Create config
- New-Item -ItemType File -Path $env:USERPROFILE/.ssh/config | Out-Null
- Add-Content -Path $env:USERPROFILE/.ssh/config -Value "StrictHostKeyChecking accept-new" -Encoding ascii
- attrib.exe +r $env:USERPROFILE/.ssh/config
variables:
GIT_STRATEGY: none
GIT_CURL_VERBOSE : 1
GIT_TRACE : 1
GIT_TRACE_PACK_ACCESS : 1
GIT_TRACE_PACKET : 1
GIT_TRACE_PACKFILE : 1
GIT_TRACE_PERFORMANCE : 1
GIT_TRACE_SETUP : 1
GIT_TRACE_SHALLOW : 1
script:
- ssh-add $env:USERPROFILE/.ssh/id_ed25519
- ssh-add -L
# OpenSSH I demand your attention to say hello to me
- ssh -V
- Get-Content C:\\Users\\vboxuser/.ssh/config
- Get-Content C:\\Users\\vboxuser/.ssh/id_ed25519
- ssh -vT -y -G git@git.fooselfhosted.com
#- ssh -T -y git@git.fooselfhosted.com
#- ssh -vT -y git@git.fooselfhosted.com
#- ssh -vvT -y git@git.fooselfhosted.com
#- ssh -vvvT git@git.fooselfhosted.com
#- ssh -v -p 22 git@git.fooselfhosted.com
#- ssh -vT -L 2222:git.fooselfhosted.com:22 git@git.fooselfhosted.com
# Detect and set the path to OpenSSH ssh.exe
- $env:GIT_SSH = Get-Command ssh.exe
- git clone --progress --verbose git@git.fooselfhosted.com:project/dir/com.foo.ugc.git
resource_group: build
when: manual
allow_failure: false
Just to explain a bit more what happens when it gets executed
# First thing I copy my ssh key from my environment variables
cp $env:SSH_KEY $env:USERPROFILE/.ssh/id_ed25519
# I create a ssh config file for the current user
New-Item -ItemType File -Path $env:USERPROFILE/.ssh/config
# I add a line to accept new connections
Add-Content -Path $env:USERPROFILE/.ssh/config -Value "StrictHostKeyChecking accept-new" -Encoding ascii
# Give some reading permissions
attrib.exe +r $env:USERPROFILE/.ssh/config
# Add the key
ssh-add $env:USERPROFILE/.ssh/id_ed25519
# List the key
ssh-add -L
# Print SSH version
ssh -V
# Prints OpenSSH_for_Windows_8.9p1, LibreSSL 3.4.2
# I tried also with the version that can be installed with Windows but same results
Get-Content C:\\Users\\vboxuser/.ssh/config
# prints what I expet the line that I added above
# StrictHostKeyChecking accept-new
Get-Content C:\\Users\\vboxuser/.ssh/id_ed25519
# Prints the correct key
# I tried to check if I could reach the host before clonning the repository
# This would correctly print a "Welcome @user" message in docker but it does happen in vbox
# I added the -G parameter to get more information
# I also tried with -v -vv and -vvv the last one does not print anything and fails right away
ssh -vT -G git@git.fooselfhosted.com
This last command prints the following
user git
hostname git.fooselfhosted.com
port 22
addressfamily any
batchmode no
canonicalizefallbacklocal yes
canonicalizehostname false
checkhostip no
compression no
controlmaster false
enablesshkeysign no
clearallforwardings no
exitonforwardfailure no
fingerprinthash SHA256
forwardx11 no
forwardx11trusted no
gatewayports no
gssapiauthentication no
gssapidelegatecredentials no
hashknownhosts no
hostbasedauthentication no
identitiesonly no
kbdinteractiveauthentication yes
nohostauthenticationforlocalhost no
passwordauthentication yes
permitlocalcommand no
proxyusefdpass no
pubkeyauthentication true
requesttty false
sessiontype default
stdinnull no
forkafterauthentication no
streamlocalbindunlink no
stricthostkeychecking accept-new
tcpkeepalive yes
tunnel false
verifyhostkeydns false
visualhostkey no
updatehostkeys true
canonicalizemaxdots 1
connectionattempts 1
forwardx11timeout 1200
numberofpasswordprompts 3
serveralivecountmax 3
serveraliveinterval 0
ciphers chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
hostkeyalgorithms ssh-ed25519-cert-v01@openssh.com,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,sk-ssh-ed25519-cert-v01@openssh.com,sk-ecdsa-sha2-nistp256-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ssh-ed25519@openssh.com,sk-ecdsa-sha2-nistp256@openssh.com,rsa-sha2-512,rsa-sha2-256
hostbasedacceptedalgorithms ssh-ed25519-cert-v01@openssh.com,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,sk-ssh-ed25519-cert-v01@openssh.com,sk-ecdsa-sha2-nistp256-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ssh-ed25519@openssh.com,sk-ecdsa-sha2-nistp256@openssh.com,rsa-sha2-512,rsa-sha2-256
kexalgorithms curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256
casignaturealgorithms ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ssh-ed25519@openssh.com,sk-ecdsa-sha2-nistp256@openssh.com,rsa-sha2-512,rsa-sha2-256
loglevel DEBUG
macs umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
securitykeyprovider internal
pubkeyacceptedalgorithms ssh-ed25519-cert-v01@openssh.com,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,sk-ssh-ed25519-cert-v01@openssh.com,sk-ecdsa-sha2-nistp256-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ssh-ed25519@openssh.com,sk-ecdsa-sha2-nistp256@openssh.com,rsa-sha2-512,rsa-sha2-256
xauthlocation /usr/X11R6/bin/xauth
identityfile ~/.ssh/id_rsa
identityfile ~/.ssh/id_ecdsa
identityfile ~/.ssh/id_ecdsa_sk
identityfile ~/.ssh/id_ed25519
identityfile ~/.ssh/id_ed25519_sk
identityfile ~/.ssh/id_xmss
identityfile ~/.ssh/id_dsa
canonicaldomains none
globalknownhostsfile __PROGRAMDATA__\ssh/ssh_known_hosts __PROGRAMDATA__\ssh/ssh_known_hosts2
userknownhostsfile C:\Users\vboxuser/.ssh/known_hosts C:\Users\vboxuser/.ssh/known_hosts2
logverbose none
permitremoteopen any
addkeystoagent false
forwardagent no
connecttimeout none
tunneldevice any:any
canonicalizePermittedcnames none
controlpersist no
escapechar ~
ipqos af21 cs1
rekeylimit 0 0
streamlocalbindmask 0177
syslogfacility USER
This parameters are exactly the same that if I log into the running VM through ssh and try to execute it manually with the same command, but I do get a Welcome message.
This is my gitlab runner configuration file
concurrent = 1
check_interval = 0
shutdown_timeout = 0
log_level = "debug"
[session_server]
session_timeout = 1800
[[runners]]
name = "VirtualBoxLaptop"
url = "https://git.fooselfhosted.com/"
id = 13
token = "XXXXX"
token_obtained_at = 2023-12-27T12:50:45Z
token_expires_at = 0001-01-01T00:00:00Z
executor = "virtualbox"
shell = "pwsh"
output_limit = 20000
builds_dir = "e:/workload/"
[runners.ssh]
user = "vboxuser"
password = "test"
port = "22"
disable_strict_host_key_checking = true
[runners.virtualbox]
base_name = "Win10-22H2-BaseBox"
allowed_images = [".*"]
base_folder = ""
disable_snapshots = true
start_type = "gui"
Important Note
Since the runner hangs I usually can have the time to log into the VM through ssh just like the gitlab-runner service probably is doing and I can execute SSH and clone the repository without any issues. I also tried with SSH -vT -G git@git.fooselfhoste.com and it works and the parameters are exactly the same I compared the output with beyond compare and they are exactly the same no differences.
Maybe I’m missing something… maybe the ports could be blocked
I also find very strange that if I use -v o -vv in ssh I don get any output and if I use -vvv it fail right away without logs. This doesn’t happen if I log into the VMs with SSH and try to do the same then it behaves as expected.
Thanks for taking the time, any ideas on how to troubleshoot or workaround the issue will help! 