Gitlab Runner - spinning up EC2 instances with docker machine fails with certificate issue

GitLab CI/CD
1

I followed these instructions to the letter: Autoscaling GitLab Runner on AWS EC2 | GitLab

I am using latest Ubuntu images.

When trying to spawn the EC2 instance, I get the error below about being unable to copy the certificate to the remote machine.

Feb 02 12:48:08 ip-172-31-29-130 gitlab-runner[3247]: Creating CA: /root/.docker/machine/certs/ca.pem     driver=amazonec2 name=runner-agarazmy-gitlab-yocto-builder-1675342087-77cc8ab9 operation=create
Feb 02 12:48:08 ip-172-31-29-130 gitlab-runner[3247]: Creating client certificate: /root/.docker/machine/certs/cert.pem  driver=amazonec2 name=runner-agarazmy-gitlab-yocto-builder-1675342087-77cc8ab9 operation=create
Feb 02 12:48:08 ip-172-31-29-130 gitlab-runner[3247]: Running pre-create checks...                        driver=amazonec2 name=runner-agarazmy-gitlab-yocto-builder-1675342087-77cc8ab9 operation=create
Feb 02 12:48:08 ip-172-31-29-130 gitlab-runner[3247]: Creating machine...                                 driver=amazonec2 name=runner-agarazmy-gitlab-yocto-builder-1675342087-77cc8ab9 operation=create
Feb 02 12:48:08 ip-172-31-29-130 gitlab-runner[3247]: (runner-agarazmy-gitlab-yocto-builder-1675342087-77cc8ab9) Launching instance...  driver=amazonec2 name=runner-agarazmy-gitlab-yocto-builder-1675342087-77cc8ab9 operation=create
Feb 02 12:48:09 ip-172-31-29-130 gitlab-runner[3247]: (runner-agarazmy-gitlab-yocto-builder-1675342087-77cc8ab9) Waiting for spot instance...  driver=amazonec2 name=runner-agarazmy-gitlab-yocto-builder-1675342087-77cc8ab9 operation=create
Feb 02 12:48:10 ip-172-31-29-130 gitlab-runner[3247]: IdleCount is set to 0 so the machine will be created on demand in job context  creating=1 idle=0 idleCount=0 idleCountMin=0 idleScaleFactor=0 maxMachineCreate=0 maxMachines=0 removing=0 runner=AgarazMy total=1 used=0
Feb 02 12:48:11 ip-172-31-29-130 gitlab-runner[3247]: Appending trace to coordinator...ok                 code=202 job=3703549791 job-log=0-229 job-status=running runner=AgarazMy sent-log=0-228 status=202 Accepted update-interval=1m0s
Feb 02 12:48:13 ip-172-31-29-130 gitlab-runner[3247]: IdleCount is set to 0 so the machine will be created on demand in job context  creating=1 idle=0 idleCount=0 idleCountMin=0 idleScaleFactor=0 maxMachineCreate=0 maxMachines=0 removing=0 runner=AgarazMy total=1 used=0
Feb 02 12:48:16 ip-172-31-29-130 gitlab-runner[3247]: IdleCount is set to 0 so the machine will be created on demand in job context  creating=1 idle=0 idleCount=0 idleCountMin=0 idleScaleFactor=0 maxMachineCreate=0 maxMachines=0 removing=0 runner=AgarazMy total=1 used=0
Feb 02 12:48:19 ip-172-31-29-130 gitlab-runner[3247]: IdleCount is set to 0 so the machine will be created on demand in job context  creating=1 idle=0 idleCount=0 idleCountMin=0 idleScaleFactor=0 maxMachineCreate=0 maxMachines=0 removing=0 runner=AgarazMy total=1 used=0
Feb 02 12:48:22 ip-172-31-29-130 gitlab-runner[3247]: IdleCount is set to 0 so the machine will be created on demand in job context  creating=1 idle=0 idleCount=0 idleCountMin=0 idleScaleFactor=0 maxMachineCreate=0 maxMachines=0 removing=0 runner=AgarazMy total=1 used=0
Feb 02 12:48:25 ip-172-31-29-130 gitlab-runner[3247]: (runner-agarazmy-gitlab-yocto-builder-1675342087-77cc8ab9) Created spot instance request sir-x66egn8h  driver=amazonec2 name=runner-agarazmy-gitlab-yocto-builder-1675342087-77cc8ab9 operation=create
Feb 02 12:48:25 ip-172-31-29-130 gitlab-runner[3247]: IdleCount is set to 0 so the machine will be created on demand in job context  creating=1 idle=0 idleCount=0 idleCountMin=0 idleScaleFactor=0 maxMachineCreate=0 maxMachines=0 removing=0 runner=AgarazMy total=1 used=0
Feb 02 12:48:25 ip-172-31-29-130 gitlab-runner[3247]: Waiting for machine to be running, this may take a few minutes...  driver=amazonec2 name=runner-agarazmy-gitlab-yocto-builder-1675342087-77cc8ab9 operation=create
Feb 02 12:48:25 ip-172-31-29-130 gitlab-runner[3247]: Detecting operating system of created instance...   driver=amazonec2 name=runner-agarazmy-gitlab-yocto-builder-1675342087-77cc8ab9 operation=create
Feb 02 12:48:25 ip-172-31-29-130 gitlab-runner[3247]: Waiting for SSH to be available...                  driver=amazonec2 name=runner-agarazmy-gitlab-yocto-builder-1675342087-77cc8ab9 operation=create
Feb 02 12:48:28 ip-172-31-29-130 gitlab-runner[3247]: IdleCount is set to 0 so the machine will be created on demand in job context  creating=1 idle=0 idleCount=0 idleCountMin=0 idleScaleFactor=0 maxMachineCreate=0 maxMachines=0 removing=0 runner=AgarazMy total=1 used=0
Feb 02 12:48:31 ip-172-31-29-130 gitlab-runner[3247]: IdleCount is set to 0 so the machine will be created on demand in job context  creating=1 idle=0 idleCount=0 idleCountMin=0 idleScaleFactor=0 maxMachineCreate=0 maxMachines=0 removing=0 runner=AgarazMy total=1 used=0
Feb 02 12:48:34 ip-172-31-29-130 gitlab-runner[3247]: IdleCount is set to 0 so the machine will be created on demand in job context  creating=1 idle=0 idleCount=0 idleCountMin=0 idleScaleFactor=0 maxMachineCreate=0 maxMachines=0 removing=0 runner=AgarazMy total=1 used=0
Feb 02 12:48:35 ip-172-31-29-130 gitlab-runner[3247]: Detecting the provisioner...                        driver=amazonec2 name=runner-agarazmy-gitlab-yocto-builder-1675342087-77cc8ab9 operation=create
Feb 02 12:48:36 ip-172-31-29-130 gitlab-runner[3247]: Provisioning with ubuntu(systemd)...                driver=amazonec2 name=runner-agarazmy-gitlab-yocto-builder-1675342087-77cc8ab9 operation=create
Feb 02 12:48:37 ip-172-31-29-130 gitlab-runner[3247]: IdleCount is set to 0 so the machine will be created on demand in job context  creating=1 idle=0 idleCount=0 idleCountMin=0 idleScaleFactor=0 maxMachineCreate=0 maxMachines=0 removing=0 runner=AgarazMy total=1 used=0
Feb 02 12:48:40 ip-172-31-29-130 gitlab-runner[3247]: IdleCount is set to 0 so the machine will be created on demand in job context  creating=1 idle=0 idleCount=0 idleCountMin=0 idleScaleFactor=0 maxMachineCreate=0 maxMachines=0 removing=0 runner=AgarazMy total=1 used=0
Feb 02 12:48:43 ip-172-31-29-130 gitlab-runner[3247]: Installing Docker...                                driver=amazonec2 name=runner-agarazmy-gitlab-yocto-builder-1675342087-77cc8ab9 operation=create
Feb 02 12:48:43 ip-172-31-29-130 gitlab-runner[3247]: IdleCount is set to 0 so the machine will be created on demand in job context  creating=1 idle=0 idleCount=0 idleCountMin=0 idleScaleFactor=0 maxMachineCreate=0 maxMachines=0 removing=0 runner=AgarazMy total=1 used=0
Feb 02 12:48:46 ip-172-31-29-130 gitlab-runner[3247]: IdleCount is set to 0 so the machine will be created on demand in job context  creating=1 idle=0 idleCount=0 idleCountMin=0 idleScaleFactor=0 maxMachineCreate=0 maxMachines=0 removing=0 runner=AgarazMy total=1 used=0
Feb 02 12:48:49 ip-172-31-29-130 gitlab-runner[3247]: IdleCount is set to 0 so the machine will be created on demand in job context  creating=1 idle=0 idleCount=0 idleCountMin=0 idleScaleFactor=0 maxMachineCreate=0 maxMachines=0 removing=0 runner=AgarazMy total=1 used=0
Feb 02 12:48:52 ip-172-31-29-130 gitlab-runner[3247]: IdleCount is set to 0 so the machine will be created on demand in job context  creating=1 idle=0 idleCount=0 idleCountMin=0 idleScaleFactor=0 maxMachineCreate=0 maxMachines=0 removing=0 runner=AgarazMy total=1 used=0
Feb 02 12:48:55 ip-172-31-29-130 gitlab-runner[3247]: IdleCount is set to 0 so the machine will be created on demand in job context  creating=1 idle=0 idleCount=0 idleCountMin=0 idleScaleFactor=0 maxMachineCreate=0 maxMachines=0 removing=0 runner=AgarazMy total=1 used=0
Feb 02 12:48:58 ip-172-31-29-130 gitlab-runner[3247]: IdleCount is set to 0 so the machine will be created on demand in job context  creating=1 idle=0 idleCount=0 idleCountMin=0 idleScaleFactor=0 maxMachineCreate=0 maxMachines=0 removing=0 runner=AgarazMy total=1 used=0
Feb 02 12:49:00 ip-172-31-29-130 gitlab-runner[3247]: Checking for jobs...nothing                         runner=AgarazMy
Feb 02 12:49:01 ip-172-31-29-130 gitlab-runner[3247]: IdleCount is set to 0 so the machine will be created on demand in job context  creating=1 idle=0 idleCount=0 idleCountMin=0 idleScaleFactor=0 maxMachineCreate=0 maxMachines=0 removing=0 runner=AgarazMy total=1 used=0
Feb 02 12:49:02 ip-172-31-29-130 gitlab-runner[3247]: Copying certs to the local machine directory...     driver=amazonec2 name=runner-agarazmy-gitlab-yocto-builder-1675342087-77cc8ab9 operation=create
Feb 02 12:49:03 ip-172-31-29-130 gitlab-runner[3247]: Copying certs to the remote machine...              driver=amazonec2 name=runner-agarazmy-gitlab-yocto-builder-1675342087-77cc8ab9 operation=create
Feb 02 12:49:04 ip-172-31-29-130 gitlab-runner[3247]: IdleCount is set to 0 so the machine will be created on demand in job context  creating=1 idle=0 idleCount=0 idleCountMin=0 idleScaleFactor=0 maxMachineCreate=0 maxMachines=0 removing=0 runner=AgarazMy total=1 used=0
Feb 02 12:49:05 ip-172-31-29-130 gitlab-runner[3247]: ERROR: Error creating machine: Error running provisioning: ssh command error:  driver=amazonec2 name=runner-agarazmy-gitlab-yocto-builder-1675342087-77cc8ab9 operation=create
Feb 02 12:49:05 ip-172-31-29-130 gitlab-runner[3247]: ERROR: command : printf '%s' '-----BEGIN CERTIFICATE-----  driver=amazonec2 name=runner-agarazmy-gitlab-yocto-builder-1675342087-77cc8ab9 operation=create
Feb 02 12:49:05 ip-172-31-29-130 gitlab-runner[3247]: ERROR: MIIC0TCCAbmgAwIBAgIQJO+lnmKwQcyWfRBOHiNriTANBgkqhkiG9w0BAQsFADAS  driver=amazonec2 name=runner-agarazmy-gitlab-yocto-builder-1675342087-77cc8ab9 operation=create
Feb 02 12:49:05 ip-172-31-29-130 gitlab-runner[3247]: ERROR: MRAwDgYDVQQKEwd1bmtub3duMB4XDTIzMDIwMjEyNDMwMFoXDTI2MDExNzEyNDMw  driver=amazonec2 name=runner-agarazmy-gitlab-yocto-builder-1675342087-77cc8ab9 operation=create
Feb 02 12:49:05 ip-172-31-29-130 gitlab-runner[3247]: ERROR: MFowEjEQMA4GA1UEChMHdW5rbm93bjCCASIwDQYJKoZIhvcNAQEBBQADggEPADCC  driver=amazonec2 name=runner-agarazmy-gitlab-yocto-builder-1675342087-77cc8ab9 operation=create
Feb 02 12:49:05 ip-172-31-29-130 gitlab-runner[3247]: ERROR: AQoCggEBAMBf6sC5sdLCPuQ5GG0VW3fGl+NPXn7kcA6Yq69ThcsvcKVPkBJzYkgB  driver=amazonec2 name=runner-agarazmy-gitlab-yocto-builder-1675342087-77cc8ab9 operation=create
Feb 02 12:49:05 ip-172-31-29-130 gitlab-runner[3247]: ERROR: Bt7YZX7ofgXQe2VnCVK3iXhFlfFIZxQXSHaGyZrKpiVZRE7EXYsraLPPX3sJKcs2  driver=amazonec2 name=runner-agarazmy-gitlab-yocto-builder-1675342087-77cc8ab9 operation=create
Feb 02 12:49:05 ip-172-31-29-130 gitlab-runner[3247]: ERROR: oDpZb7LS8z9PZO08t5X0t0cqcahCKcmYak4vjMEEjKG4/zvxnpHLxMhUf9bovV4C  driver=amazonec2 name=runner-agarazmy-gitlab-yocto-builder-1675342087-77cc8ab9 operation=create
Feb 02 12:49:05 ip-172-31-29-130 gitlab-runner[3247]: ERROR: rTwdQN3Oz4ZTxTZ8dBwIOyDdGZBIJQtZc2tVIo6JFWfdQ0Kx0prg1qsP+lBL01no  driver=amazonec2 name=runner-agarazmy-gitlab-yocto-builder-1675342087-77cc8ab9 operation=create
Feb 02 12:49:05 ip-172-31-29-130 gitlab-runner[3247]: ERROR: 89zjAWX95v3m3zghXUA3AuBiCzUWz15RqhgIZLK3EOb9+5wkmbeSsQ/H8ewYucSQ  driver=amazonec2 name=runner-agarazmy-gitlab-yocto-builder-1675342087-77cc8ab9 operation=create
Feb 02 12:49:05 ip-172-31-29-130 gitlab-runner[3247]: ERROR: L6qX7ZruKVdI1nOd218qXzcHK04WW6UCAwEAAaMjMCEwDgYDVR0PAQH/BAQDAgKs  driver=amazonec2 name=runner-agarazmy-gitlab-yocto-builder-1675342087-77cc8ab9 operation=create
Feb 02 12:49:05 ip-172-31-29-130 gitlab-runner[3247]: ERROR: MA8GA1UdEwEB/wQFMAMBAf8wDQYJKoZIhvcNAQELBQADggEBABP+riks/UEP8o43  driver=amazonec2 name=runner-agarazmy-gitlab-yocto-builder-1675342087-77cc8ab9 operation=create
Feb 02 12:49:05 ip-172-31-29-130 gitlab-runner[3247]: ERROR: QcC1y7peief2OK/pw37VcH2c9E/zNGHjPMhUT/DfA+3j8kluPTY6oCaz6QZ8M9T7  driver=amazonec2 name=runner-agarazmy-gitlab-yocto-builder-1675342087-77cc8ab9 operation=create
Feb 02 12:49:05 ip-172-31-29-130 gitlab-runner[3247]: ERROR: aS6d/NSal5BeJ9TuJkeybnLr2QvjTYrk9t6EsbOKobW2D6cOvVbeh92qbiPsKkr5  driver=amazonec2 name=runner-agarazmy-gitlab-yocto-builder-1675342087-77cc8ab9 operation=create
Feb 02 12:49:05 ip-172-31-29-130 gitlab-runner[3247]: ERROR: 7h+f/0LMI4BPs7cJeDodEEFWnDIcfMfMuNVyxSVjr4Os1D0FiX0vYwDXQ6e/BPGX  driver=amazonec2 name=runner-agarazmy-gitlab-yocto-builder-1675342087-77cc8ab9 operation=create
Feb 02 12:49:05 ip-172-31-29-130 gitlab-runner[3247]: ERROR: KXu7pHY21ZZzKGFbunnkKZlrlVxl/2onuyxdKEto51ma84U5mJQ4WyjZA4IbBzBF  driver=amazonec2 name=runner-agarazmy-gitlab-yocto-builder-1675342087-77cc8ab9 operation=create
Feb 02 12:49:05 ip-172-31-29-130 gitlab-runner[3247]: ERROR: y+dP7TS2YFivlAGDj51J7kAyaNczx7v8p40bpy/wjmWgOk9/fCu3kcpkIRxilHlb  driver=amazonec2 name=runner-agarazmy-gitlab-yocto-builder-1675342087-77cc8ab9 operation=create
Feb 02 12:49:05 ip-172-31-29-130 gitlab-runner[3247]: ERROR: 8aeaNy4=                                     driver=amazonec2 name=runner-agarazmy-gitlab-yocto-builder-1675342087-77cc8ab9 operation=create
Feb 02 12:49:05 ip-172-31-29-130 gitlab-runner[3247]: ERROR: -----END CERTIFICATE-----                    driver=amazonec2 name=runner-agarazmy-gitlab-yocto-builder-1675342087-77cc8ab9 operation=create
Feb 02 12:49:05 ip-172-31-29-130 gitlab-runner[3247]: ERROR: ' | sudo tee /etc/docker/ca.pem              driver=amazonec2 name=runner-agarazmy-gitlab-yocto-builder-1675342087-77cc8ab9 operation=create
Feb 02 12:49:05 ip-172-31-29-130 gitlab-runner[3247]: ERROR: err     : exit status 1                      driver=amazonec2 name=runner-agarazmy-gitlab-yocto-builder-1675342087-77cc8ab9 operation=create
Feb 02 12:49:05 ip-172-31-29-130 gitlab-runner[3247]: ERROR: output  : -----BEGIN CERTIFICATE-----        driver=amazonec2 name=runner-agarazmy-gitlab-yocto-builder-1675342087-77cc8ab9 operation=create
Feb 02 12:49:05 ip-172-31-29-130 gitlab-runner[3247]: ERROR: MIIC0TCCAbmgAwIBAgIQJO+lnmKwQcyWfRBOHiNriTANBgkqhkiG9w0BAQsFADAS  driver=amazonec2 name=runner-agarazmy-gitlab-yocto-builder-1675342087-77cc8ab9 operation=create
Feb 02 12:49:05 ip-172-31-29-130 gitlab-runner[3247]: ERROR: MRAwDgYDVQQKEwd1bmtub3duMB4XDTIzMDIwMjEyNDMwMFoXDTI2MDExNzEyNDMw  driver=amazonec2 name=runner-agarazmy-gitlab-yocto-builder-1675342087-77cc8ab9 operation=create
Feb 02 12:49:05 ip-172-31-29-130 gitlab-runner[3247]: ERROR: MFowEjEQMA4GA1UEChMHdW5rbm93bjCCASIwDQYJKoZIhvcNAQEBBQADggEPADCC  driver=amazonec2 name=runner-agarazmy-gitlab-yocto-builder-1675342087-77cc8ab9 operation=create
Feb 02 12:49:05 ip-172-31-29-130 gitlab-runner[3247]: ERROR: AQoCggEBAMBf6sC5sdLCPuQ5GG0VW3fGl+NPXn7kcA6Yq69ThcsvcKVPkBJzYkgB  driver=amazonec2 name=runner-agarazmy-gitlab-yocto-builder-1675342087-77cc8ab9 operation=create
Feb 02 12:49:05 ip-172-31-29-130 gitlab-runner[3247]: ERROR: Bt7YZX7ofgXQe2VnCVK3iXhFlfFIZxQXSHaGyZrKpiVZRE7EXYsraLPPX3sJKcs2  driver=amazonec2 name=runner-agarazmy-gitlab-yocto-builder-1675342087-77cc8ab9 operation=create
Feb 02 12:49:05 ip-172-31-29-130 gitlab-runner[3247]: ERROR: oDpZb7LS8z9PZO08t5X0t0cqcahCKcmYak4vjMEEjKG4/zvxnpHLxMhUf9bovV4C  driver=amazonec2 name=runner-agarazmy-gitlab-yocto-builder-1675342087-77cc8ab9 operation=create
Feb 02 12:49:05 ip-172-31-29-130 gitlab-runner[3247]: ERROR: rTwdQN3Oz4ZTxTZ8dBwIOyDdGZBIJQtZc2tVIo6JFWfdQ0Kx0prg1qsP+lBL01no  driver=amazonec2 name=runner-agarazmy-gitlab-yocto-builder-1675342087-77cc8ab9 operation=create
Feb 02 12:49:05 ip-172-31-29-130 gitlab-runner[3247]: ERROR: 89zjAWX95v3m3zghXUA3AuBiCzUWz15RqhgIZLK3EOb9+5wkmbeSsQ/H8ewYucSQ  driver=amazonec2 name=runner-agarazmy-gitlab-yocto-builder-1675342087-77cc8ab9 operation=create
Feb 02 12:49:05 ip-172-31-29-130 gitlab-runner[3247]: ERROR: L6qX7ZruKVdI1nOd218qXzcHK04WW6UCAwEAAaMjMCEwDgYDVR0PAQH/BAQDAgKs  driver=amazonec2 name=runner-agarazmy-gitlab-yocto-builder-1675342087-77cc8ab9 operation=create
Feb 02 12:49:05 ip-172-31-29-130 gitlab-runner[3247]: ERROR: MA8GA1UdEwEB/wQFMAMBAf8wDQYJKoZIhvcNAQELBQADggEBABP+riks/UEP8o43  driver=amazonec2 name=runner-agarazmy-gitlab-yocto-builder-1675342087-77cc8ab9 operation=create
Feb 02 12:49:05 ip-172-31-29-130 gitlab-runner[3247]: ERROR: QcC1y7peief2OK/pw37VcH2c9E/zNGHjPMhUT/DfA+3j8kluPTY6oCaz6QZ8M9T7  driver=amazonec2 name=runner-agarazmy-gitlab-yocto-builder-1675342087-77cc8ab9 operation=create
Feb 02 12:49:05 ip-172-31-29-130 gitlab-runner[3247]: ERROR: aS6d/NSal5BeJ9TuJkeybnLr2QvjTYrk9t6EsbOKobW2D6cOvVbeh92qbiPsKkr5  driver=amazonec2 name=runner-agarazmy-gitlab-yocto-builder-1675342087-77cc8ab9 operation=create
Feb 02 12:49:05 ip-172-31-29-130 gitlab-runner[3247]: ERROR: 7h+f/0LMI4BPs7cJeDodEEFWnDIcfMfMuNVyxSVjr4Os1D0FiX0vYwDXQ6e/BPGX  driver=amazonec2 name=runner-agarazmy-gitlab-yocto-builder-1675342087-77cc8ab9 operation=create
Feb 02 12:49:05 ip-172-31-29-130 gitlab-runner[3247]: ERROR: KXu7pHY21ZZzKGFbunnkKZlrlVxl/2onuyxdKEto51ma84U5mJQ4WyjZA4IbBzBF  driver=amazonec2 name=runner-agarazmy-gitlab-yocto-builder-1675342087-77cc8ab9 operation=create
Feb 02 12:49:05 ip-172-31-29-130 gitlab-runner[3247]: ERROR: y+dP7TS2YFivlAGDj51J7kAyaNczx7v8p40bpy/wjmWgOk9/fCu3kcpkIRxilHlb  driver=amazonec2 name=runner-agarazmy-gitlab-yocto-builder-1675342087-77cc8ab9 operation=create
Feb 02 12:49:05 ip-172-31-29-130 gitlab-runner[3247]: ERROR: 8aeaNy4=                                     driver=amazonec2 name=runner-agarazmy-gitlab-yocto-builder-1675342087-77cc8ab9 operation=create
Feb 02 12:49:05 ip-172-31-29-130 gitlab-runner[3247]: ERROR: -----END CERTIFICATE-----                    driver=amazonec2 name=runner-agarazmy-gitlab-yocto-builder-1675342087-77cc8ab9 operation=create
Feb 02 12:49:05 ip-172-31-29-130 gitlab-runner[3247]: ERROR: tee: /etc/docker/ca.pem: No such file or directory  driver=amazonec2 name=runner-agarazmy-gitlab-yocto-builder-1675342087-77cc8ab9 operation=create
Feb 02 12:49:05 ip-172-31-29-130 gitlab-runner[3247]: WARNING: Problem while reading command output       error=read |0: file already closed
Feb 02 12:49:05 ip-172-31-29-130 gitlab-runner[3247]: WARNING: Problem while reading command output       error=read |0: file already closed
Feb 02 12:49:05 ip-172-31-29-130 gitlab-runner[3247]: ERROR: Machine creation failed                      error=exit status 1 name=runner-agarazmy-gitlab-yocto-builder-1675342087-77cc8ab9 time=57.713797101s
Feb 02 12:49:05 ip-172-31-29-130 gitlab-runner[3247]: WARNING: Requesting machine removal                 lifetime=57.714009243s name=runner-agarazmy-gitlab-yocto-builder-1675342087-77cc8ab9 now=2023-02-02 12:49:05.593906193 +0000 UTC m=+343.458655762 reason=Failed to create used=57.714010208s usedCount=0
Feb 02 12:49:05 ip-172-31-29-130 gitlab-runner[3247]: WARNING: Stopping machine                           lifetime=57.736810392s name=runner-agarazmy-gitlab-yocto-builder-1675342087-77cc8ab9 reason=Failed to create used=22.619984ms usedCount=0
Feb 02 12:49:05 ip-172-31-29-130 gitlab-runner[3247]: Stopping "runner-agarazmy-gitlab-yocto-builder-1675342087-77cc8ab9"...  name=runner-agarazmy-gitlab-yocto-builder-1675342087-77cc8ab9 operation=stop
Feb 02 12:49:05 ip-172-31-29-130 gitlab-runner[3247]: ERROR: UnsupportedOperation: You can't stop the Spot Instance 'i-09cb5ed90c4d41d24' because it is associated with a one-time Spot Instance request. You can only stop Spot Instances associated with persistent Spot Instance requests.  name=runner-agarazmy-gitlab-yocto-builder-1675342087-77cc8ab9 operation=stop
Feb 02 12:49:05 ip-172-31-29-130 gitlab-runner[3247]: ERROR:         status code: 400, request id: bc37ae69-e035-4ae0-8b59-b28b6728d8e1  name=runner-agarazmy-gitlab-yocto-builder-1675342087-77cc8ab9 operation=stop
Feb 02 12:49:05 ip-172-31-29-130 gitlab-runner[3247]: WARNING: Problem while reading command output       error=read |0: file already closed
Feb 02 12:49:05 ip-172-31-29-130 gitlab-runner[3247]: WARNING: Error while stopping machine               error=exit status 1 lifetime=58.032750367s name=runner-agarazmy-gitlab-yocto-builder-1675342087-77cc8ab9 reason=Failed to create used=318.559573ms usedCount=0
Feb 02 12:49:05 ip-172-31-29-130 gitlab-runner[3247]: WARNING: Removing machine                           lifetime=58.032915849s name=runner-agarazmy-gitlab-yocto-builder-1675342087-77cc8ab9 reason=Failed to create used=318.72474ms usedCount=0
Feb 02 12:49:05 ip-172-31-29-130 gitlab-runner[3247]: About to remove runner-agarazmy-gitlab-yocto-builder-1675342087-77cc8ab9  name=runner-agarazmy-gitlab-yocto-builder-1675342087-77cc8ab9 operation=remove
Feb 02 12:49:05 ip-172-31-29-130 gitlab-runner[3247]: WARNING: This action will delete both local reference and remote instance.  name=runner-agarazmy-gitlab-yocto-builder-1675342087-77cc8ab9 operation=remove
Feb 02 12:49:06 ip-172-31-29-130 gitlab-runner[3247]: Successfully removed runner-agarazmy-gitlab-yocto-builder-1675342087-77cc8ab9  name=runner-agarazmy-gitlab-yocto-builder-1675342087-77cc8ab9 operation=remove
Feb 02 12:49:06 ip-172-31-29-130 gitlab-runner[3247]: WARNING: Problem while reading command output       error=read |0: file already closed
Feb 02 12:49:06 ip-172-31-29-130 gitlab-runner[3247]: WARNING: Problem while reading command output       error=read |0: file already closed
Feb 02 12:49:06 ip-172-31-29-130 gitlab-runner[3247]: Machine removed                                     lifetime=58.530313407s name=runner-agarazmy-gitlab-yocto-builder-1675342087-77cc8ab9 now=2023-02-02 12:49:06.410211575 +0000 UTC m=+344.274961181 reason=Failed to create retries=0 used=816.122648ms usedCount=0
Feb 02 12:49:06 ip-172-31-29-130 gitlab-runner[3247]: Running pre-create checks...                        driver=amazonec2 name=runner-agarazmy-gitlab-yocto-builder-1675342146-55255e76 operation=create
Feb 02 12:49:06 ip-172-31-29-130 gitlab-runner[3247]: Creating machine...                                 driver=amazonec2 name=runner-agarazmy-gitlab-yocto-builder-1675342146-55255e76 operation=create

Anyone else seeing anything like this?

2 Likes
2

Yes! Have you managed to resolve this?

1 Like
3

No, do you know when it started? We haven’t run this particular build script since the 27th November 2022 so something must have changed between then.

I tried to rebuild the infra from scratch and it made absolutely no difference.

I also just tried copying the certificate from /root/.docker/machine/certs/ as I saw a log line about creating certificate there but that also made no difference.

Anyone else?

4

It suddenly started happening after runner manager happily creating instances for a long time. SO not sure !

5

I’m going to try spinning it up on Debian instead and see if that makes any difference.

6

It started happening about 2 hours ago, have never seen it before…

7

Same problem here

Version: 15.7.2
Git revision: 0e7679e6
Git branch: 15-7-stable
GO version: go1.18.9
Built: 2023-01-13T23:55:39+0000
OS/Arch: linux/amd64

8

andreylh Did you also start having this issue today?

9

Yes, for me it also started about 2 hours ago

10

Are you both using Ubuntu as the host OS for the gitlab runner coordinator? What version?

11

Created an issue here Certificate issue on Ubuntu when using docker machine (#29593) · Issues · GitLab.org / gitlab-runner · GitLab

2 Likes
12

Yes, Ubuntu 20.04.5 LTS

13

Hi! Same problem here. I read the posts for issue #29593 shared by @jwithecombe, and fixed the issue by following the @raldred suggestion and including the machine option:
“engine-install-url=https://releases.rancher.com/install-docker/20.10.sh”

14

Hi there, a docker change recently broke our builds so maybe try this fix?