SSH AWS EC2 instance from gitlab runner timeout intermittently

GitLab CI/CD
, , , ,
1

Hi, I am using gitlab runner and do ssh to AWS EC2 instance to install the node application. However I am getting timeout error intermittently.

ssh: connect to host X.X.X.X port 22: Connection timed out

Here is the way I SSH to the instance in gitlab-ci file
$ ssh -i $SSH_PRIVATE_KEY -o StrictHostKeyChecking=no $SSH_USER@$QA_EC2_IPADDRESS

I went through various documents and articles but didn’t find any such straight forward solution.

Could anyone please here?

2

Maybe the EC2 instance has a firewall policy in place, blocking external traffic via port 22/SSH. I’d suggest verifying, and adding an allow policy for the source IP address of your GitLab runner host(s).

3

Many thanks @dnsmichi I have added the source IP of gitlab runner which is obviously the public IP of EC2 instance as a inbound rule with 22/SSH. Still facing the intermittent error.

4

Not sure I understand the source => destination flow.

  • GitLab Runner is installed in an EC2 instance, lets call it ec2-1-runner
  • The deployment happens on a 2nd EC2 instance, lets call it ec2-2-deploy
  • ec2-1-runner uses ssh to connect to ec2-2-deploy
  • ec2-2-deploy has an inbound firewall rule for tcp/22 from ec2-1-runner

Is that assumption correct?

5

@dnsmichi, Yes, exactly the same settings I have. Thank you.

6

Thanks for confirming. For testing, I’d suggest you SSH into the ec2-1-runner VM, and try to verify the connection to the ec2-2-deploy VM by using manual CLI tools - nmap for scanning open ports, and ssh to test the connection.

nmap -Pn -A ec2-2-deploy

ssh -vvvv ec2-2-deploy

Please post your results here, and make sure to replace the hostnames/IP addresses with the names above.

If the connections do not work, verify the inbound firewall rules in your AWS EC2 console. Maybe the source IP address is wrong, or the network range does not match (e.g. /30 is too limiting and needs /28 or the like).

7

Apologies for my misunderstanding. So here is the flow what I have.

  • GitLab Runner is installed in an EC2 instance, lets call it ec2-1-runner
  • The deployment happens on a same EC2 instance, ec2-1-runner
  • ec2-1-runner uses ssh to connect to ec2-1-runner
  • ec2-1-runner has an inbound firewall rule for tcp/22 from ec2-1-runner

Basically, I have both gitlab runner and deployment happening at same instance. Please guide.

8

Finally I got to know what’s the problem. When I was trying deploy, it picks any one of the shared runner which was not added as an inbound rules. So to fix this, I have added a special tag in .gitlab-ci.yml file like below and it picks all the deployment

  tags:
    - ci-wiredplus-api
1 Like