Struggling with docker and systemd?

How to Use GitLab
,
1

Hi All,

I have been implementing the use of Hashicorp Packer and Virtualbox to simultaneously create AWS AMIs and Hashicorp Vagrant boxes - the former for our staging and production environments and the latter for development. The concept being to use the same build process to create both dev and prod and so keep them in sync - no surprises!

I have been able to successfully run our code handraulically but struggled to get our self-hosted Gitlab-Runner instance (running on a dedicated server) to run a pipeline using the same code. Our runner was setup to use a docker+machine executor and I was unable to get Virtualbox (essential to building Vagrant boxes) to install as it requires systemd which does not play well with docker (as I discovered).

After 2 weeks of dedicated but ultimately fruitless effort trying to shoehorn systemd into a docker image I finally came to the conclusion that another solution was needed - one which didn’t feel like trying to take off wheel nuts with a pair of pliers!

As we were already using Virtualbox to host the docker+machine executor on the Runner server I settled on the addition of a Virtualbox executor to the Runner. That way I could use a dedicated Ubuntu VM as my runner and all would be well…right? Well, almost. We do have a working solution but there were a few more wrinkles than had been originally expected.

The AMIs we require can still be generated correctly by Packer using the docker-machine executor (so I continue to use that for this purpose) and I can now generate the Vagrant boxes using Packer on the Virtualbox executor and separate these two calls from the same pipeline by tagging the runner with “systemd” and marking up the portion of the gitlab-ci.yml file that requires systemd with that tag.

Happy to take questions
Stay safe

Regards
Ian Carson

Here’s the solution as text and as a (hopefully) helpful diagram.

VM Executor Solution
VM Executor Solution2361Ă—1992 95.9 KB

The VirtualBox executor calls upon a VM on the dedicated gitlab-runner server where I have VirtualBox installed. Config.toml section shown below (you can ignore the cache section - we run a minio server for caching)

[[runners]]
    name = "ubuntu-base"
    url = "https://gitlab.paradigm.local"
    token = "REDACTED"
    tls-ca-file = "REDACTED"
    executor = "virtualbox"
    [runners.custom_build_dir]
    [runners.cache]
        Type = "s3"
        [runners.cache.s3]
          ServerAddress = "runner01.paradigm.local"
          AccessKey = "REDACTED"
          SecretKey = "REDACTED"
          BucketName = "REDACTED"
      [runners.cache.gcs]
    [runners.ssh]
        user = "REDACTED"
        password = "REDACTED"
    [runners.virtualbox]
        base_name = "ubuntu-18-04-packer-vbox"
        disable_snapshots = false

The virtualbox executor VM has Packer and VirtualBox installed and this nested VirtualBox instance has a base VM (plain Ubuntu 18.04) from which I can use Packer’s virtualbox-vm builder and various provisioners to create an appropriately provisioned VM. Our gitlab-ci.yml looks like this (the before script is just a test to ensure access to the right resources is available to the build)

stages:
  - build

before_script:
  - packer --version
  - vboxmanage -v
  - cat /etc/os-release

build:packer-virtualbox:
  stage: build

  tags:
    - systemd

  script:
    - packer build packer-vm-test.json

I can then create the Vagrant box from this created VM using a Packer post-processor. A simplified packer-vm-test.json looks like this (note the use of environment variables in the variables section - these come from the project’s CI/CD variables in Gitlab. We use Gitlab to provide a single source of truth for all versioning so that version changes percolate through to all builds)

{
	"variables": {
		"headless": "true",
		"hashicorp_release_url": "{{ env `HASHICORP_RELEASE_URL` }}",
		"packer_version": "{{env `PACKER_VERSION`}}",
		"vagrant_group": "vagrant",
		"vagrant_user": "vagrant",
		"vagrant_comment": "Vagrant",
		"vagrant_home": "/home/vagrant"
	},
	"builders": [
		{
			"type": "virtualbox-vm",
			"communicator": "ssh",
			"headless": "{{user `headless`}}",
			"ssh_username": "redacted",
			"ssh_password": "redacted",
			"ssh_timeout": "1h",
			"guest_additions_mode": "disable",
			"shutdown_command": "sudo shutdown -h now",
			"output_directory": "./builds-vm",
			"vm_name": "ubuntu-18-04-nested",
			"boot_wait": "10s",
			"virtualbox_version_file": "/home/ianc/.vbox_version",
			"attach_snapshot": "Base_State",
			"target_snapshot": "ubuntu-18.04-vagrant-user-box",
			"force_delete_snapshot": "true",
			"keep_registered": "false",
			"skip_export": "false"
		}
	],
	"provisioners": [
		{
			"type": "file",
			"source": "./scripts",
			"destination": "/tmp",
			"pause_before": "50ms"
		},
		{
			"type": "shell",
			"environment_vars": [
				"GROUP={{ user `vagrant_group` }}",
				"USER={{ user `vagrant_user` }}",
				"COMMENT={{ user `vagrant_comment` }}",
				"HOME={{ user `vagrant_home` }}"
			],
			"inline_shebang": "/bin/bash -eux",
			"inline": ["/tmp/scripts/setup-user.sh"]
		}
	],
	"post-processors": [
		[
			{
				"type": "vagrant",
				"output": "./builds-vm/ubuntu-18-04-vagrant-user-box-test2.box"
			},
			{
				"type": "shell-local",
				"inline_shebang": "/bin/bash -eux",
				"inline": [
					"scp -i ~/.ssh/id_rsa ./builds-vm/ubuntu-18-04-vagrant-user-box-test2.box ian@paradigmgpu:ubuntu-18-04-vagrant-user-box-test2.box"
				]
			}
		]
	]
}

Artifacts from the Packer Vagrant box creation process are saved onto the executor VM and copied from there to a self-hosted Vagrant repository by running shell scripts called by Packer - see the post processors section of the above JSON file. This self-hosted repository is not directly germane to the solution and is included here for completeness

1 Like