Job failed (system failure): prepare environment: waiting for pod running: timed out waiting for pod to start

Describe your question in as much detail as possible:
Sometimes my gitlab runner in the K8s cluster waits too much, but no resources is found during this time, so the job fails with the message:

Job failed (system failure): prepare environment: waiting for pod running: timed out waiting for pod to start
  1. How to extend the pending for pod time?
  2. Is there any better way to trigger the pod only when resources are available?
  • What are you seeing, and how does that differ from what you expect to see?
e[0KRunning with gitlab-runner 16.3.0 (8ec04662)e[0;m
e[0K  on gitlab-runner-large-5fff54fc46-x5f8q 9yTHc-qn, system ID: r_j7leLNFWuqNCe[0;m
e[0K  feature flags: FF_USE_ADVANCED_POD_SPEC_CONFIGURATION:truee[0;m
section_start:1699275041:prepare_executor
e[0Ke[0Ke[36;1mPreparing the "kubernetes" executore[0;me[0;m
e[0KUsing Kubernetes namespace: gitlab-runnere[0;m
e[0KUsing Kubernetes executor with image cubinary.dc.local:8084/algo:v22.05-pt1.12-1 ...e[0;m
e[0KUsing attach strategy to execute scripts...e[0;m
section_end:1699275041:prepare_executor
e[0Ksection_start:1699275041:prepare_script
e[0Ke[0Ke[36;1mPreparing environmente[0;me[0;m
e[0;33mWARNING: Advanced Pod Spec configuration enabled, merging the provided PodSpec to the generated one. This is an alpha feature and is subject to change. Feedback is collected in this issue: https://gitlab.com/gitlab-org/gitlab-runner/-/issues/29659 ...e[0;m
Waiting for pod gitlab-runner/runner-9ythc-qn-project-142-concurrent-2-u9o1czzj to be running, status is Pending
Waiting for pod gitlab-runner/runner-9ythc-qn-project-142-concurrent-2-u9o1czzj to be running, status is Pending
	ContainersNotReady: "containers with unready status: [build helper]"
	ContainersNotReady: "containers with unready status: [build helper]"
Waiting for pod gitlab-runner/runner-9ythc-qn-project-142-concurrent-2-u9o1czzj to be running, status is Pending
	ContainersNotReady: "containers with unready status: [build helper]"
	ContainersNotReady: "containers with unready status: [build helper]"
Waiting for pod gitlab-runner/runner-9ythc-qn-project-142-concurrent-2-u9o1czzj to be running, status is Pending
	ContainersNotReady: "containers with unready status: [build helper]"
	ContainersNotReady: "containers with unready status: [build helper]"
Waiting for pod gitlab-runner/runner-9ythc-qn-project-142-concurrent-2-u9o1czzj to be running, status is Pending
	ContainersNotReady: "containers with unready status: [build helper]"
	ContainersNotReady: "containers with unready status: [build helper]"
Waiting for pod gitlab-runner/runner-9ythc-qn-project-142-concurrent-2-u9o1czzj to be running, status is Pending
	ContainersNotReady: "containers with unready status: [build helper]"
	ContainersNotReady: "containers with unready status: [build helper]"
Waiting for pod gitlab-runner/runner-9ythc-qn-project-142-concurrent-2-u9o1czzj to be running, status is Pending
	ContainersNotReady: "containers with unready status: [build helper]"
	ContainersNotReady: "containers with unready status: [build helper]"
Waiting for pod gitlab-runner/runner-9ythc-qn-project-142-concurrent-2-u9o1czzj to be running, status is Pending
	ContainersNotReady: "containers with unready status: [build helper]"
	ContainersNotReady: "containers with unready status: [build helper]"
Waiting for pod gitlab-runner/runner-9ythc-qn-project-142-concurrent-2-u9o1czzj to be running, status is Pending
	ContainersNotReady: "containers with unready status: [build helper]"
	ContainersNotReady: "containers with unready status: [build helper]"
Waiting for pod gitlab-runner/runner-9ythc-qn-project-142-concurrent-2-u9o1czzj to be running, status is Pending
	ContainersNotReady: "containers with unready status: [build helper]"
	ContainersNotReady: "containers with unready status: [build helper]"
Waiting for pod gitlab-runner/runner-9ythc-qn-project-142-concurrent-2-u9o1czzj to be running, status is Pending
	ContainersNotReady: "containers with unready status: [build helper]"
	ContainersNotReady: "containers with unready status: [build helper]"
Waiting for pod gitlab-runner/runner-9ythc-qn-project-142-concurrent-2-u9o1czzj to be running, status is Pending
	ContainersNotReady: "containers with unready status: [build helper]"
	ContainersNotReady: "containers with unready status: [build helper]"
Waiting for pod gitlab-runner/runner-9ythc-qn-project-142-concurrent-2-u9o1czzj to be running, status is Pending
	ContainersNotReady: "containers with unready status: [build helper]"
	ContainersNotReady: "containers with unready status: [build helper]"
Waiting for pod gitlab-runner/runner-9ythc-qn-project-142-concurrent-2-u9o1czzj to be running, status is Pending
	ContainersNotReady: "containers with unready status: [build helper]"
	ContainersNotReady: "containers with unready status: [build helper]"
Waiting for pod gitlab-runner/runner-9ythc-qn-project-142-concurrent-2-u9o1czzj to be running, status is Pending
	ContainersNotReady: "containers with unready status: [build helper]"
	ContainersNotReady: "containers with unready status: [build helper]"
Waiting for pod gitlab-runner/runner-9ythc-qn-project-142-concurrent-2-u9o1czzj to be running, status is Pending
	ContainersNotReady: "containers with unready status: [build helper]"
	ContainersNotReady: "containers with unready status: [build helper]"
Waiting for pod gitlab-runner/runner-9ythc-qn-project-142-concurrent-2-u9o1czzj to be running, status is Pending
	ContainersNotReady: "containers with unready status: [build helper]"
	ContainersNotReady: "containers with unready status: [build helper]"
Waiting for pod gitlab-runner/runner-9ythc-qn-project-142-concurrent-2-u9o1czzj to be running, status is Pending
	ContainersNotReady: "containers with unready status: [build helper]"
	ContainersNotReady: "containers with unready status: [build helper]"
Waiting for pod gitlab-runner/runner-9ythc-qn-project-142-concurrent-2-u9o1czzj to be running, status is Pending
	ContainersNotReady: "containers with unready status: [build helper]"
	ContainersNotReady: "containers with unready status: [build helper]"
Waiting for pod gitlab-runner/runner-9ythc-qn-project-142-concurrent-2-u9o1czzj to be running, status is Pending
	ContainersNotReady: "containers with unready status: [build helper]"
	ContainersNotReady: "containers with unready status: [build helper]"
Waiting for pod gitlab-runner/runner-9ythc-qn-project-142-concurrent-2-u9o1czzj to be running, status is Pending
	ContainersNotReady: "containers with unready status: [build helper]"
	ContainersNotReady: "containers with unready status: [build helper]"
Waiting for pod gitlab-runner/runner-9ythc-qn-project-142-concurrent-2-u9o1czzj to be running, status is Pending
	ContainersNotReady: "containers with unready status: [build helper]"
	ContainersNotReady: "containers with unready status: [build helper]"
Waiting for pod gitlab-runner/runner-9ythc-qn-project-142-concurrent-2-u9o1czzj to be running, status is Pending
	ContainersNotReady: "containers with unready status: [build helper]"
	ContainersNotReady: "containers with unready status: [build helper]"
Waiting for pod gitlab-runner/runner-9ythc-qn-project-142-concurrent-2-u9o1czzj to be running, status is Pending
	ContainersNotReady: "containers with unready status: [build helper]"
	ContainersNotReady: "containers with unready status: [build helper]"
Waiting for pod gitlab-runner/runner-9ythc-qn-project-142-concurrent-2-u9o1czzj to be running, status is Pending
	ContainersNotReady: "containers with unready status: [build helper]"
	ContainersNotReady: "containers with unready status: [build helper]"
Waiting for pod gitlab-runner/runner-9ythc-qn-project-142-concurrent-2-u9o1czzj to be running, status is Pending
	ContainersNotReady: "containers with unready status: [build helper]"
	ContainersNotReady: "containers with unready status: [build helper]"
Waiting for pod gitlab-runner/runner-9ythc-qn-project-142-concurrent-2-u9o1czzj to be running, status is Pending
	ContainersNotReady: "containers with unready status: [build helper]"
	ContainersNotReady: "containers with unready status: [build helper]"
Waiting for pod gitlab-runner/runner-9ythc-qn-project-142-concurrent-2-u9o1czzj to be running, status is Pending
	ContainersNotReady: "containers with unready status: [build helper]"
	ContainersNotReady: "containers with unready status: [build helper]"
Waiting for pod gitlab-runner/runner-9ythc-qn-project-142-concurrent-2-u9o1czzj to be running, status is Pending
	ContainersNotReady: "containers with unready status: [build helper]"
	ContainersNotReady: "containers with unready status: [build helper]"
Waiting for pod gitlab-runner/runner-9ythc-qn-project-142-concurrent-2-u9o1czzj to be running, status is Pending
	ContainersNotReady: "containers with unready status: [build helper]"
	ContainersNotReady: "containers with unready status: [build helper]"
Waiting for pod gitlab-runner/runner-9ythc-qn-project-142-concurrent-2-u9o1czzj to be running, status is Pending
	ContainersNotReady: "containers with unready status: [build helper]"
	ContainersNotReady: "containers with unready status: [build helper]"
Waiting for pod gitlab-runner/runner-9ythc-qn-project-142-concurrent-2-u9o1czzj to be running, status is Pending
	ContainersNotReady: "containers with unready status: [build helper]"
	ContainersNotReady: "containers with unready status: [build helper]"
Waiting for pod gitlab-runner/runner-9ythc-qn-project-142-concurrent-2-u9o1czzj to be running, status is Pending
	ContainersNotReady: "containers with unready status: [build helper]"
	ContainersNotReady: "containers with unready status: [build helper]"
Waiting for pod gitlab-runner/runner-9ythc-qn-project-142-concurrent-2-u9o1czzj to be running, status is Pending
	ContainersNotReady: "containers with unready status: [build helper]"
	ContainersNotReady: "containers with unready status: [build helper]"
Waiting for pod gitlab-runner/runner-9ythc-qn-project-142-concurrent-2-u9o1czzj to be running, status is Pending
	ContainersNotReady: "containers with unready status: [build helper]"
	ContainersNotReady: "containers with unready status: [build helper]"
Waiting for pod gitlab-runner/runner-9ythc-qn-project-142-concurrent-2-u9o1czzj to be running, status is Pending
	ContainersNotReady: "containers with unready status: [build helper]"
	ContainersNotReady: "containers with unready status: [build helper]"
Waiting for pod gitlab-runner/runner-9ythc-qn-project-142-concurrent-2-u9o1czzj to be running, status is Pending
	ContainersNotReady: "containers with unready status: [build helper]"
	ContainersNotReady: "containers with unready status: [build helper]"
Waiting for pod gitlab-runner/runner-9ythc-qn-project-142-concurrent-2-u9o1czzj to be running, status is Pending
	ContainersNotReady: "containers with unready status: [build helper]"
	ContainersNotReady: "containers with unready status: [build helper]"
Waiting for pod gitlab-runner/runner-9ythc-qn-project-142-concurrent-2-u9o1czzj to be running, status is Pending
	ContainersNotReady: "containers with unready status: [build helper]"
	ContainersNotReady: "containers with unready status: [build helper]"
Waiting for pod gitlab-runner/runner-9ythc-qn-project-142-concurrent-2-u9o1czzj to be running, status is Pending
	ContainersNotReady: "containers with unready status: [build helper]"
	ContainersNotReady: "containers with unready status: [build helper]"
Waiting for pod gitlab-runner/runner-9ythc-qn-project-142-concurrent-2-u9o1czzj to be running, status is Pending
	ContainersNotReady: "containers with unready status: [build helper]"
	ContainersNotReady: "containers with unready status: [build helper]"
Waiting for pod gitlab-runner/runner-9ythc-qn-project-142-concurrent-2-u9o1czzj to be running, status is Pending
	ContainersNotReady: "containers with unready status: [build helper]"
	ContainersNotReady: "containers with unready status: [build helper]"
Waiting for pod gitlab-runner/runner-9ythc-qn-project-142-concurrent-2-u9o1czzj to be running, status is Pending
	ContainersNotReady: "containers with unready status: [build helper]"
	ContainersNotReady: "containers with unready status: [build helper]"
Waiting for pod gitlab-runner/runner-9ythc-qn-project-142-concurrent-2-u9o1czzj to be running, status is Pending
	ContainersNotReady: "containers with unready status: [build helper]"
	ContainersNotReady: "containers with unready status: [build helper]"
Waiting for pod gitlab-runner/runner-9ythc-qn-project-142-concurrent-2-u9o1czzj to be running, status is Pending
	ContainersNotReady: "containers with unready status: [build helper]"
	ContainersNotReady: "containers with unready status: [build helper]"
Waiting for pod gitlab-runner/runner-9ythc-qn-project-142-concurrent-2-u9o1czzj to be running, status is Pending
	ContainersNotReady: "containers with unready status: [build helper]"
	ContainersNotReady: "containers with unready status: [build helper]"
Waiting for pod gitlab-runner/runner-9ythc-qn-project-142-concurrent-2-u9o1czzj to be running, status is Pending
	ContainersNotReady: "containers with unready status: [build helper]"
	ContainersNotReady: "containers with unready status: [build helper]"
Waiting for pod gitlab-runner/runner-9ythc-qn-project-142-concurrent-2-u9o1czzj to be running, status is Pending
	ContainersNotReady: "containers with unready status: [build helper]"
	ContainersNotReady: "containers with unready status: [build helper]"
Waiting for pod gitlab-runner/runner-9ythc-qn-project-142-concurrent-2-u9o1czzj to be running, status is Pending
	ContainersNotReady: "containers with unready status: [build helper]"
	ContainersNotReady: "containers with unready status: [build helper]"
Waiting for pod gitlab-runner/runner-9ythc-qn-project-142-concurrent-2-u9o1czzj to be running, status is Pending
	ContainersNotReady: "containers with unready status: [build helper]"
	ContainersNotReady: "containers with unready status: [build helper]"
Waiting for pod gitlab-runner/runner-9ythc-qn-project-142-concurrent-2-u9o1czzj to be running, status is Pending
	ContainersNotReady: "containers with unready status: [build helper]"
	ContainersNotReady: "containers with unready status: [build helper]"
Waiting for pod gitlab-runner/runner-9ythc-qn-project-142-concurrent-2-u9o1czzj to be running, status is Pending
	ContainersNotReady: "containers with unready status: [build helper]"
	ContainersNotReady: "containers with unready status: [build helper]"
Waiting for pod gitlab-runner/runner-9ythc-qn-project-142-concurrent-2-u9o1czzj to be running, status is Pending
	ContainersNotReady: "containers with unready status: [build helper]"
	ContainersNotReady: "containers with unready status: [build helper]"
Waiting for pod gitlab-runner/runner-9ythc-qn-project-142-concurrent-2-u9o1czzj to be running, status is Pending
	ContainersNotReady: "containers with unready status: [build helper]"
	ContainersNotReady: "containers with unready status: [build helper]"
Waiting for pod gitlab-runner/runner-9ythc-qn-project-142-concurrent-2-u9o1czzj to be running, status is Pending
	ContainersNotReady: "containers with unready status: [build helper]"
	ContainersNotReady: "containers with unready status: [build helper]"
Waiting for pod gitlab-runner/runner-9ythc-qn-project-142-concurrent-2-u9o1czzj to be running, status is Pending
	ContainersNotReady: "containers with unready status: [build helper]"
	ContainersNotReady: "containers with unready status: [build helper]"
Waiting for pod gitlab-runner/runner-9ythc-qn-project-142-concurrent-2-u9o1czzj to be running, status is Pending
	ContainersNotReady: "containers with unready status: [build helper]"
	ContainersNotReady: "containers with unready status: [build helper]"
Waiting for pod gitlab-runner/runner-9ythc-qn-project-142-concurrent-2-u9o1czzj to be running, status is Pending
	ContainersNotReady: "containers with unready status: [build helper]"
	ContainersNotReady: "containers with unready status: [build helper]"
Waiting for pod gitlab-runner/runner-9ythc-qn-project-142-concurrent-2-u9o1czzj to be running, status is Pending
	ContainersNotReady: "containers with unready status: [build helper]"
	ContainersNotReady: "containers with unready status: [build helper]"
Waiting for pod gitlab-runner/runner-9ythc-qn-project-142-concurrent-2-u9o1czzj to be running, status is Pending
	ContainersNotReady: "containers with unready status: [build helper]"
	ContainersNotReady: "containers with unready status: [build helper]"
Waiting for pod gitlab-runner/runner-9ythc-qn-project-142-concurrent-2-u9o1czzj to be running, status is Pending
	ContainersNotReady: "containers with unready status: [build helper]"
	ContainersNotReady: "containers with unready status: [build helper]"
section_end:1699275225:prepare_script
e[0Ke[31;1mERROR: Job failed (system failure): prepare environment: waiting for pod running: timed out waiting for pod to start. Check https://docs.gitlab.com/runner/shells/index.html#shell-profile-loading for more informatione[0;m

  • Consider including screenshots, error messages, and/or other helpful visuals

  • What version are you on? Are you using self-managed or GitLab.com? self-managed

    • GitLab (Hint: /help): 14.6.1-ee
    • Runner (Hint: /admin/runners): gitlab-runner:alpine-v16.3.0
  • What troubleshooting steps have you already taken? Can you link to any docs or other resources so we know where you have been?
    We are just hitting retry every time until resources are available (lack of resources)

Here is the config section in my values.yaml:

config: |
      [[runners]]
        environment = ["DOCKER_HOST=tcp://docker:2376", 
                        "DOCKER_TLS_CERTDIR=/certs",
                        "DOCKER_TLS_VERIFY=1",
                        "DOCKER_CERT_PATH=$DOCKER_TLS_CERTDIR/client",
                        "FF_USE_ADVANCED_POD_SPEC_CONFIGURATION=true"
                      ]
        [runners.kubernetes]
          namespace = "{{.Release.Namespace}}"
          image = "ubuntu:22.04"
          privileged = true
          [runners.kubernetes.affinity]
              [runners.kubernetes.affinity.node_affinity]
                [[runners.kubernetes.affinity.node_affinity.preferred_during_scheduling_ignored_during_execution]]
                  weight = 100
                  [runners.kubernetes.affinity.node_affinity.preferred_during_scheduling_ignored_during_execution.preference]
                    [[runners.kubernetes.affinity.node_affinity.preferred_during_scheduling_ignored_during_execution.preference.match_expressions]]
                      key = "gitlab-runner-generic"
                      operator = "In"
                      values = ["true"]
          [runners.kubernetes.node_tolerations]
            "gitlab-runner-generic=true" = "NoSchedule"
          [[runners.kubernetes.pod_spec]]
            name = "cpu-2c4g"
            patch = '''
              [
                  {
                    "op": "add",
                    "path": "/containers/0/resources",
                    "value": {
                      "limits": {
                        "memory": "4Gi",
                        "cpu": 2
                      },
                      "requests": {
                        "memory": "1Gi",
                        "cpu": 1
                      }
                    }
                  }
              ]
            '''
            patch_type = "json"
        [[runners.kubernetes.volumes.host_path]]
          name = "releases"
          mount_path = "/Releases"
          read_only = false
          host_path = "/Releases"             
        [[runners.kubernetes.volumes.empty_dir]]
          name = "docker-certs"
          mount_path = "/certs/client"
          medium = "Memory"
        [[runners.kubernetes.volumes.config_map]]
          name = "daemon-config"
          mount_path = "/etc/docker/"
          [runners.kubernetes.volumes.config_map.items]
            "daemon-cpu" = "daemon.json"
        [[runners.kubernetes.volumes.secret]]
           name = "gitlab-cert"
           mount_path = "/home/gitlab-runner/.gitlab-runner/certs/gitlab.---.---.local.crt"

the crt name censored

1 Like