Job failed (system failure): prepare environment

GitLab CI/CD
#1

** Job failed (system failure): prepare environment

gitlab runner is installed on our K8s cluster

CICD jobs are retrieved but fail with the following output:

  • What version are you on? Are you using self-managed or GitLab.com?

Attached debug logs from the runner:

Checking for jobs... received                       job=1868419446 repo_url=https://gitlab.com/gamestopcorp/platform/blockchain/wallet/backend.git runner=3w57Bdz_
Processing chain                                    chain-leaf=[0x40005d2000 0x40005d2b00 0x400070b600] context=certificate-chain-build
Certificate doesn't provide parent URL: exiting the loop  Issuer=USERTrust RSA Certification Authority IssuerCertURL=[] Serial=2645093764781058787591871645665788717 Subject=USERTrust RSA Certification Authority context=certificate-chain-build
Processing chain                                    chain-leaf=[0x40005d2000 0x40005d2b00 0x40005d3080 0x4000458680] context=certificate-chain-build
Certificate doesn't provide parent URL: exiting the loop  Issuer=AAA Certificate Services IssuerCertURL=[] Serial=1 Subject=AAA Certificate Services context=certificate-chain-build
Requeued the runner                                 builds=1 runner=3w57Bdz_
Running with gitlab-runner 14.5.0 (f0a95a76)        job=1868419446 project=31935927 runner=3w57Bdz_
  on gitlab-runner-gitlab-runner-6c576bc9d5-jb52j 3w57Bdz_  job=1868419446 project=31935927 runner=3w57Bdz_
Resolving secrets                       job=1868419446 project=31935927 runner=3w57Bdz_
Preparing the "kubernetes" executor     job=1868419446 project=31935927 runner=3w57Bdz_
Regex allowing overrides for Namespace is empty, disabling override.  job=1868419446 project=31935927 runner=3w57Bdz_
Regex allowing overrides for ServiceAccount is empty, disabling override.  job=1868419446 project=31935927 runner=3w57Bdz_
Regex allowing overrides for BearerToken is empty, disabling override.  job=1868419446 project=31935927 runner=3w57Bdz_
Regex allowing overrides for PodAnnotations is empty, disabling override.  job=1868419446 project=31935927 runner=3w57Bdz_
setting allowing overrides for CPURequest is empty, disabling override.  job=1868419446 project=31935927 runner=3w57Bdz_
setting allowing overrides for MemoryRequest is empty, disabling override.  job=1868419446 project=31935927 runner=3w57Bdz_
setting allowing overrides for EphemeralStorageRequest is empty, disabling override.  job=1868419446 project=31935927 runner=3w57Bdz_
setting allowing overrides for CPULimit is empty, disabling override.  job=1868419446 project=31935927 runner=3w57Bdz_
setting allowing overrides for MemoryLimit is empty, disabling override.  job=1868419446 project=31935927 runner=3w57Bdz_
setting allowing overrides for EphemeralStorageLimit is empty, disabling override.  job=1868419446 project=31935927 runner=3w57Bdz_
setting allowing overrides for ServiceCPURequest is empty, disabling override.  job=1868419446 project=31935927 runner=3w57Bdz_
setting allowing overrides for ServiceMemoryRequest is empty, disabling override.  job=1868419446 project=31935927 runner=3w57Bdz_
setting allowing overrides for ServiceEphemeralStorageRequest is empty, disabling override.  job=1868419446 project=31935927 runner=3w57Bdz_
setting allowing overrides for ServiceCPULimit is empty, disabling override.  job=1868419446 project=31935927 runner=3w57Bdz_
setting allowing overrides for ServiceMemoryLimit is empty, disabling override.  job=1868419446 project=31935927 runner=3w57Bdz_
setting allowing overrides for ServiceEphemeralStorageLimit is empty, disabling override.  job=1868419446 project=31935927 runner=3w57Bdz_
setting allowing overrides for HelperCPURequest is empty, disabling override.  job=1868419446 project=31935927 runner=3w57Bdz_
setting allowing overrides for HelperMemoryRequest is empty, disabling override.  job=1868419446 project=31935927 runner=3w57Bdz_
setting allowing overrides for HelperEphemeralStorageRequest is empty, disabling override.  job=1868419446 project=31935927 runner=3w57Bdz_
setting allowing overrides for HelperCPULimit is empty, disabling override.  job=1868419446 project=31935927 runner=3w57Bdz_
setting allowing overrides for HelperMemoryLimit is empty, disabling override.  job=1868419446 project=31935927 runner=3w57Bdz_
setting allowing overrides for HelperEphemeralStorageLimit is empty, disabling override.  job=1868419446 project=31935927 runner=3w57Bdz_
Using Kubernetes namespace: gitlab                  job=1868419446 project=31935927 runner=3w57Bdz_
Using Kubernetes executor with image registry.gitlab.com/gitlab-org/terraform-images/stable:latest ...  job=1868419446 project=31935927 runner=3w57Bdz_
Using attach strategy to execute scripts...         job=1868419446 project=31935927 runner=3w57Bdz_
Using helper image: registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-f0a95a76  job=1868419446 project=31935927 runner=3w57Bdz_
Shell configuration: environment: []
dockercommand:
- sh
- -c
- "if [ -x /usr/local/bin/bash ]; then\n\texec /usr/local/bin/bash \nelif [ -x /usr/bin/bash
  ]; then\n\texec /usr/bin/bash \nelif [ -x /bin/bash ]; then\n\texec /bin/bash \nelif
  [ -x /usr/local/bin/sh ]; then\n\texec /usr/local/bin/sh \nelif [ -x /usr/bin/sh
  ]; then\n\texec /usr/bin/sh \nelif [ -x /bin/sh ]; then\n\texec /bin/sh \nelif [
  -x /busybox/sh ]; then\n\texec /busybox/sh \nelse\n\techo shell not found\n\texit
  1\nfi\n\n"
command: bash
arguments: []
passfile: false
extension: ""
  job=1868419446 project=31935927 runner=3w57Bdz_
Waiting for signals...                              job=1868419446 project=31935927 runner=3w57Bdz_
No referees configured                              job=1868419446 project=31935927 runner=3w57Bdz_
Executing build stage                               build_stage=prepare_script job=1868419446 project=31935927 runner=3w57Bdz_
Preparing environment                   job=1868419446 project=31935927 runner=3w57Bdz_
Starting Kubernetes command with attach...          job=1868419446 project=31935927 runner=3w57Bdz_
Setting up secrets                                  job=1868419446 project=31935927 runner=3w57Bdz_
Setting up scripts config map                       job=1868419446 project=31935927 runner=3w57Bdz_
Checking for jobs... nothing                        runner=3w57Bdz_
Appending trace to coordinator... ok                code=202 job=1868419446 job-log=0-692 job-status=running runner=3w57Bdz_ sent-log=0-691 status=202 Accepted update-interval=1m0s
Feeding runners to channel                          builds=1
Feeding runners to channel                          builds=1
Failed to request job: runner requestConcurrency meet  builds=1 runner=3w57Bdz_
Dialing: tcp gitlab.com:443 ...                    
Submitting job to coordinator... ok                 code=200 job=1868419446 job-status= runner=3w57Bdz_ update-interval=0s
Checking for jobs... nothing                        runner=3w57Bdz_
Feeding runners to channel                          builds=1
Feeding runners to channel                          builds=1
Failed to request job: runner requestConcurrency meet  builds=1 runner=3w57Bdz_
Submitting job to coordinator... ok                 code=200 job=1868419446 job-status= runner=3w57Bdz_ update-interval=0s
Checking for jobs... nothing                        runner=3w57Bdz_
Feeding runners to channel                          builds=1
Feeding runners to channel                          builds=1
Failed to request job: runner requestConcurrency meet  builds=1 runner=3w57Bdz_
Submitting job to coordinator... ok                 code=200 job=1868419446 job-status= runner=3w57Bdz_ update-interval=0s
Checking for jobs... nothing                        runner=3w57Bdz_
Feeding runners to channel                          builds=1
Feeding runners to channel                          builds=1
Failed to request job: runner requestConcurrency meet  builds=1 runner=3w57Bdz_
Setting up build pod                                job=1868419446 project=31935927 runner=3w57Bdz_
DNSPolicy string is blank, using "ClusterFirst" as default 
Creating build pod                                  job=1868419446 project=31935927 runner=3w57Bdz_
Creating pod proxy services                         job=1868419446 project=31935927 runner=3w57Bdz_
Appending trace to coordinator... ok                code=202 job=1868419446 job-log=0-798 job-status=running runner=3w57Bdz_ sent-log=692-797 status=202 Accepted update-interval=1m0s
ERROR: Job failed (system failure): prepare environment: waiting for pod running: pod status is failed. Check https://docs.gitlab.com/runner/shells/index.html#shell-profile-loading for more information  duration_s=243.985650391 job=1868419446 project=31935927 runner=3w57Bdz_
Appending trace to coordinator... ok                code=202 job=1868419446 job-log=0-1054 job-status=running runner=3w57Bdz_ sent-log=798-1053 status=202 Accepted update-interval=1m0s
Submitting job to coordinator... accepted, but not yet completed  code=202 job=1868419446 job-status= runner=3w57Bdz_ update-interval=1s
Submitting job to coordinator... ok                 code=200 job=1868419446 job-status= runner=3w57Bdz_ update-interval=0s
WARNING: Failed to process runner                   builds=0 error=prepare environment: waiting for pod running: pod status is failed. Check https://docs.gitlab.com/runner/shells/index.html#shell-profile-loading for more information executor=kubernetes runner=3w57Bdz_
Checking for jobs... nothing                        runner=3w57Bdz_
Feeding runners to channel                          builds=0