Virtualbox Runner: time/date/connectivity not correct inside of cloned VM

GitLab CI/CD
1

Problem to solve

I have a number of Gitlab runners. Most are Docker but we also have 2 Virtualbox runners. One of them is Linux as the host and Linux in the VM and another if Linux as the host and Windows as the VM.

This issue happening on both Virtuabox runners and not in the Docker runners.

Occaisionally, we will get a job that goes to the Virtualbox runners that will fail due to that runner not having the correct time and date and causing it not to recognize the certificates from our apt repo.

W: Skipping acquire of configured file 'universe/cnf/Commands-all' as repository 'https://repo.example.com/repository/sw-focal focal InRelease' doesn't have the component 'universe' (component misspelt in sources.list?)
W: Skipping acquire of configured file 'universe/cnf/Commands-amd64' as repository 'https://repo.example.com/repository/sw-focal focal InRelease' doesn't have the component 'universe' (component misspelt in sources.list?)
E: Release file for http://security.ubuntu.com/ubuntu/dists/focal-security/InRelease is not valid yet (invalid for another 2d 8h 40min 18s). Updates for this repository will not be applied.
Cleaning up project directory and file based variables

If we restart the job, it works. The certificates on our servers are correct. We have checked those a few times.

Steps to reproduce

Set up a virtualbox runner with a pipeline that causes it to build a package from source using packages from Ubuntu. We only build proprietary packages so I can’t give an example of other packages.

This does not happen every time we run the pipeline. It is intermittent, but common enough to be very annoying.

Configuration

config.toml

concurrent = 1
check_interval = 0
shutdown_timeout = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "virtualbox"
  url = "https://gitlab.example.com/"
  id = 72
  token = "_immatoken"
  token_obtained_at = 2023-06-15T15:04:56Z
  token_expires_at = 0001-01-01T00:00:00Z
  executor = "virtualbox"
  [runners.cache]
    MaxUploadedArchiveSize = 0
  [runners.ssh]
    user = "gitlab-runner"
    password = "linux"
    identity_file = "/var/lib/gitlab-runner/.ssh/id_ed25519"
    disable_strict_host_key_checking = true
  [runners.virtualbox]
    base_name = "virtualbox-template"
    base_folder = ""
    disable_snapshots = false
    start_type = ""

Versions

# gitlab-runner --version
Version:      16.0.2
Git revision: 85586bd1
Git branch:   16-0-stable
GO version:   go1.19.9
Built:        2023-06-02T17:31:27+0000
OS/Arch:      linux/amd64

Please select whether options apply, and add the version information.

  • Self-managed
  • GitLab.com SaaS
  • Self-hosted Runners
2

I have upgrade to the latest version of gitlab-runner. We are still seeing the same issues in some of our overnight builds over the weekend.

3

We found more information. The time in the cloned VM is from the last time that the master VM was run. The cloned VM is NOT taking the current time from the VM host like it should.

We’ve mitigated the issue by forcing ntpd to sync at the beginning of the job. This will make sure that it is always running with the correct time before starting the build.

There is still a bug present either in the virtualbox runner plugin or in virtualbox itself. I can’t replicate this issue by just cloning the master VM manually without the runner.