Unable to pull gitlab-runner-helper on AWS Spot instance Docker Machine

Hello,

we’re autoscaling runners on AWS Spot instances using Docker Machine. I followed the steps here Autoscaling GitLab Runner on AWS EC2 | GitLab. From time to time there’s following error:

    Cleaning up file based variables
    Authenticating with credentials from job payload (GitLab Registry)
    Pulling docker image registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-775dd39d ...
    WARNING: Failed to pull image with policy "always": error during connect: Post https://172.16.3.247:2376/v1.25/images/create?fromImage=registry.gitlab.com%2Fgitlab-org%2Fgitlab-runner%2Fgitlab-runner-helper&tag=x86_64-775dd39d: dial tcp 172.16.3.247:2376: connect: no route to host (docker.go:147:3s)
    ERROR: Failed to cleanup volumes
    ERROR: Job failed (system failure): error during connect: Post https://172.16.3.247:2376/v1.25/containers/dba47ac6645e3a17e926c830c72560680a8d238c46b694a581f69982e1fc44bd/wait: dial tcp 172.16.3.247:2376: connect: no route to host

The executor is trying to pull the helper image but probably couldn’t even reach the docker host. Is it trying to reach the docker engine on the docker machine spawned by the runner? How do I set up a proper docker host when the IP address is different with each docker machine? Why is it even pulling the helper image (doesn’t seem to be needed)?

The build is running on image node:14.15.0 (no dind)

concurrent = 3
check_interval = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "GoAccount-Spot-Runner"
  url = "https://gitlab.com/"
  token = "***"
  executor = "docker+machine"
  limit = 3
  environment = ["FF_GITLAB_REGISTRY_HELPER_IMAGE=1"]
  [runners.cache]
    Type = "s3"
    Shared = true
    [runners.cache.s3]
      ServerAddress = "s3.amazonaws.com"
      AccessKey = "***"
      SecretKey = "***"
      BucketName = "goaccount-gitlab-runner-cache"
      BucketLocation = "eu-west-1"
  [runners.docker]
    image = "alpine"
    privileged = true
    disable_cache = true
  [runners.machine]
    IdleCount = 0
    IdleTime = 1800
    MaxGrowthRate = 1
    MaxBuilds = 10
    MachineDriver = "amazonec2"
    MachineName = "gitlab-docker-machine-%s"
    MachineOptions = [
      "amazonec2-access-key=***",
      "amazonec2-secret-key=***",
      "amazonec2-region=eu-west-1",
      "amazonec2-vpc-id=vpc-0407ce7c645d90bd4",
      "amazonec2-subnet-id=subnet-060924a544c96e6b4",
      "amazonec2-zone=c",
      "amazonec2-use-private-address=true",
      "amazonec2-security-group=staging-goaccount-gitlab-runner",
      "amazonec2-request-spot-instance=true",
      "amazonec2-spot-price=0.2",
      "amazonec2-instance-type=c5.xlarge",
      "amazonec2-block-duration-minutes=60"
    ]

Thanks