I have gitlab runners v0.70.3 on my EKS 1.31 with AZ autoscaling on us-east-1a,b,c.
Sometimes during pipeline execution appearing pretty strange error:
Log from runner agent:
Submitting job to coordinator...ok bytesize=5926 checksum=crc32:1cefcb38 code=200 job=8520994771 job-status=running runner=8uar6Nz_w update-interval=0s
WARNING: Error streaming logs gitlab-build-runner/runner-8uar6nzw-project-32243361-concurrent-0-60ok8xmc/helper:/logs-32243361-8520994771/output.log: command terminated with exit code 137. Retrying... job=8520994771 project=32243361 runner=8uar6Nz_w
WARNING: Error while executing file based variables removal script error=pod status is failed job=8520994771 project=32243361 runner=8uar6Nz_w
ERROR: Job failed (system failure): pod "runner-8uar6nzw-project-32243361-concurrent-0-60ok8xmc" status is "Failed" duration_s=178.026626479 job=8520994771 project=32243361 runner=8uar6Nz_w
Of course I have check MEM metrics for job pod, limits were not exceeded.
Also I have check settings of pipeline, no health checks or something similar which could be reason of error.
This error is unpredictable cuz In major cases all my ci jobs passing successfully but 1/10 with that error.