Running into issues with Autoscaling EC2 CI/CD Group Runners

I’m currently running into an issue where the gitlab-runner-manager EC2 that I setup is throwing this error:

 Missing instance ID, this is likely due to a failure during machine creation

My config is as follows (with certain things redacted):

concurrent = 10
check_interval = 0

  session_timeout = 1800

  name = "gitlab-runner-manager"
  url = ""
  token = "REDACTED"
  executor = "docker+machine"
  limit = 10
    Type = "s3"
    Shared = true
	ServerAddress = ""
	AccessKey = "REDACTED"
	SecretKey = "REDACTED"
	BucketName = "REDACTED"
	BucketLocation = "us-east-1"
    IdleCount = 1
    IdleTime = 1800
    MachineDriver = "amazonec2"
    MachineName = "gitlab-docker-machine-%s"
    MachineOptions = [
      Periods = ["* * * * * * *"]
      IdleCount = 1
      IdleTime = 3600
      Timezone = "UTC"

Other details that are important:

  • The docker-machine-scaler security group has port 22 and docker port 2376 traffic allowed from the gitlab-runner-manager ec2.
  • The runner version running on the gitlab-runner-manager ec2 is 14.9.1.
  • Followed the documentation here: Autoscaling GitLab Runner on AWS EC2 | GitLab But wondering if there are other required variables that need to be set in the runners.machine.MachineOptions.
  • Currently using not the enterprise hosted omnibus version

Would appreciate any help! Thanks in advance!

What Linux version are you using? The docker-machine can fail if the fork of docker-machine relies on default 16.04 ubuntu, which docker no longer supports. The fix in this case is to upgrade to the latest beta of docker-machine maintained by gitlab ([drivers/amazonec2/amazonec2.go · main · / Ops Sub-Department / docker-machine · GitLab])
After this, the output of watch docker-machine ls would sometimes still present the error but then correct itself after about a minute
Hope this helps!