[solved]Gitlab runner with fargate prepare environment failed

Hello,

I try to run gitlab-runner with AWS fargate.

The ci-coordinator task is launch properly, the docker image is pulled from my ECR without issue. But when it runs i got the following errror

Running with gitlab-runner 13.12.0 (7a6612da) on XXXX Preparing the “custom” executor
INFO[2021-05-28T11:12:28Z] Starting fargate PID=11256 version=“0.2.0 (933d940)”
INFO[2021-05-28T11:12:28Z] Executing the command PID=11256 command=config_exec
Using Custom executor with driver fargate 0.2.0 (933d940)…
INFO[2021-05-28T11:12:28Z] Starting fargate PID=11262 version=“0.2.0 (933d940)”
INFO[2021-05-28T11:12:28Z] Executing the command PID=11262 command=prepare_exec
INFO[2021-05-28T11:12:33Z] Starting new Fargate task PID=11262 command=prepare_exec
INFO[2021-05-28T11:12:33Z] Persisting data that will be used by other commands PID=11262 command=prepare_exec taskARN=“arn:aws:ecs:eu-XXXXXb”
INFO[2021-05-28T11:12:33Z] Waiting Fargate task to be ready PID=11262 command=prepare_exec taskARN=“arn:aws:ecs:eu-XXXXb”
INFO[2021-05-28T11:13:04Z] Persisting data that will be used by other commands PID=11262 command=prepare_exec taskARN=“arn:aws:ecs:eu-XXXXXb”
Preparing environment
INFO[2021-05-28T11:13:04Z] Starting fargate PID=11275 version=“0.2.0 (933d940)”
INFO[2021-05-28T11:13:04Z] Executing the command PID=11275 command=run_exec stage=prepare_script
INFO[2021-05-28T11:13:04Z] Reading file content PID=11275 command=run_exec file=/tmp/custom-executor064852123/script120423742/script. stage=prepare_script
INFO[2021-05-28T11:13:04Z] Fetching task data from metadata storage PID=11275 command=run_exec stage=prepare_script
INFO[2021-05-28T11:13:04Z] Executing script in the task container PID=11275 command=run_exec stage=prepare_script taskARN=“arn:aws:ecXXXb”
ERRO[2021-05-28T11:15:13Z] Application execution failed PID=11275 error=“executing the script on the remote host: executing script on container with IP "A.B.C.D": connecting to server: connecting to server "A.B.C.D:22" as user "root": dial tcp A.B.C.D: connect: connection timed out”
ERROR: Job failed (system failure): prepare environment: exit status 2. Check Types of shells supported by GitLab Runner | GitLab for more information`

note :

  • I have a docker configuration that run in EC smoothly and work perfectly
  • And I have a second runner that use fargate.
  • the security group allow to do ssh call from ec2 with gitlab-runner to the security group provided in the fargate executor config file. And from task to task.
  • to validate that it was not my docker that give the error, I try the debian one provided in the documentation of fargate excutor configuration. With the exact same result.

What I observe, I didn’t see any new task except the ci-coordinator poped in the ecs cluster

Is there some specific policy, port or other to apply or open?
thanks

solved by myself, after reading the code of the fargate excutor.

=>
In the driver configuration, the parameter EnablePublicIP means that it will use the public ip of the container instead of the private to run ssh command.
So I just put it to false, to use a private ip and then the security group that I provided works perfectly.

Thanks so much for figuring this out. I was banging my ahead against the wall with the same problem.

I wish i would helped me as well, but with public ip enabled false i get no working network any more:

on RUNNER_1654346466 jSc9acLh

[3](https://gitlab.com/MyCloud/group/service-tests/-/jobs/2546743457#L3)Resolving secrets00:00

[5](https://gitlab.com/MyCloud/group/service-tests/-/jobs/2546743457#L5)Preparing the "custom" executor

[6](https://gitlab.com/MyCloud/group/service-tests/-/jobs/2546743457#L6)INFO[2022-06-04T12:41:19Z] Starting fargate PID=25 version="0.1.0-beta-10-g79e1255 (79e1255)"

[7](https://gitlab.com/MyCloud/group/service-tests/-/jobs/2546743457#L7)INFO[2022-06-04T12:41:19Z] Executing the command PID=25 command=config_exec

[8](https://gitlab.com/MyCloud/group/service-tests/-/jobs/2546743457#L8)Using Custom executor with driver fargate 0.1.0-beta-10-g79e1255 (79e1255)...

[9](https://gitlab.com/MyCloud/group/service-tests/-/jobs/2546743457#L9)INFO[2022-06-04T12:41:19Z] Starting fargate PID=31 version="0.1.0-beta-10-g79e1255 (79e1255)"

[10](https://gitlab.com/MyCloud/group/service-tests/-/jobs/2546743457#L10)INFO[2022-06-04T12:41:19Z] Executing the command PID=31 command=prepare_exec

[11](https://gitlab.com/MyCloud/group/service-tests/-/jobs/2546743457#L11)INFO[2022-06-04T12:41:24Z] Starting new Fargate task PID=31 command=prepare_exec

[12](https://gitlab.com/MyCloud/group/service-tests/-/jobs/2546743457#L12)INFO[2022-06-04T12:41:24Z] Persisting data that will be used by other commands PID=31 command=prepare_exec taskARN="arn:aws:ecs:eu-central-1:033405355422:task/playground-ecs/36ed77e2f813439c8790dbb9301d407f"

[13](https://gitlab.com/MyCloud/group/service-tests/-/jobs/2546743457#L13)INFO[2022-06-04T12:41:24Z] Waiting Fargate task to be ready PID=31 command=prepare_exec taskARN="arn:aws:ecs:eu-central-1:033405355422:task/playground-ecs/36ed77e2f813439c8790dbb9301d407f"

[14](https://gitlab.com/MyCloud/group/service-tests/-/jobs/2546743457#L14)ERRO[2022-06-04T12:43:31Z] Error when waiting for task initialization. Will stop the task for cleanup PID=31 command=prepare_exec error="waiting for Fargate task to be in running state: error waiting AWS Fargate Task \"arn:aws:ecs:eu-central-1:033405355422:task/playground-ecs/36ed77e2f813439c8790dbb9301d407f\" to be in \"Running\" state: ResourceNotReady: failed waiting for successful resource state" taskARN="arn:aws:ecs:eu-central-1:033405355422:task/playground-ecs/36ed77e2f813439c8790dbb9301d407f"

[15](https://gitlab.com/MyCloud/group/service-tests/-/jobs/2546743457#L15)ERRO[2022-06-04T12:43:31Z] Application execution failed PID=31 error="waiting Fargate task to be ready: waiting for Fargate task to be in running state: error waiting AWS Fargate Task \"arn:aws:ecs:eu-central-1:033405355422:task/playground-ecs/36ed77e2f813439c8790dbb9301d407f\" to be in \"Running\" state: ResourceNotReady: failed waiting for successful resource state"

[16](https://gitlab.com/MyCloud/group/service-tests/-/jobs/2546743457#L16)Preparing environment00:00

[17](https://gitlab.com/MyCloud/group/service-tests/-/jobs/2546743457#L17)INFO[2022-06-04T12:43:36Z] Starting fargate PID=43 version="0.1.0-beta-10-g79e1255 (79e1255)"

[18](https://gitlab.com/MyCloud/group/service-tests/-/jobs/2546743457#L18)INFO[2022-06-04T12:43:36Z] Executing the command PID=43 command=run_exec stage=prepare_script

[19](https://gitlab.com/MyCloud/group/service-tests/-/jobs/2546743457#L19)INFO[2022-06-04T12:43:36Z] Reading file content PID=43 command=run_exec file=/tmp/custom-executor3893058600/script156442815/script. stage=prepare_script

[20](https://gitlab.com/MyCloud/group/service-tests/-/jobs/2546743457#L20)INFO[2022-06-04T12:43:36Z] Fetching task data from metadata storage PID=43 command=run_exec stage=prepare_script

[21](https://gitlab.com/MyCloud/group/service-tests/-/jobs/2546743457#L21)INFO[2022-06-04T12:43:36Z] Executing script in the task container PID=43 command=run_exec stage=prepare_script taskARN="arn:aws:ecs:eu-central-1:033405355422:task/playground-ecs/36ed77e2f813439c8790dbb9301d407f"

[22](https://gitlab.com/MyCloud/group/service-tests/-/jobs/2546743457#L22)ERRO[2022-06-04T12:43:36Z] Application execution failed PID=43 error="executing the script on the remote host: executing script on container with IP \"\": connecting to server: connecting to server \":22\" as user \"root\": dial tcp :22: connect: connection refused"

[23](https://gitlab.com/MyCloud/group/service-tests/-/jobs/2546743457#L23)ERROR: Job failed (system failure): prepare environment: exit status 2. Check https://docs.gitlab.com/runner/shells/index.html#shell-profile-loading for more information