Simple way to verify connection runner -> gitlab.com

How to Use GitLab
1

Hello,

we have a runner behind a firewall. Although connections are allowed, the runner is not accessing gitlab.com for some reasons.

Therefore we are asking for help on how to investigate/how to simply test connection gitlab.com ↔ runner.

In the logs I have :

adf2 gitlab-runner[21125]: #033[0;33mWARNING: Checking for jobs… failed #033[0;m #033[0;33mrunner#033[0;m=0472a85c #033[0;33mstatus#033[0;m=couldn’t execute POST against https://gitlab.com/api/v4/jobs/request: Post https://gitlab.com/api/v4/jobs/request: read tcp 172.20.103.24:35902->35.231.145.151:443: read: connection timed out
Jul 15 19:57:43 adf2 gitlab-runner: #033[0;33mWARNING: Checking for jobs… failed #033[0;m #033[0;33mrunner#033[0;m=0472a85c #033[0;33mstatus#033[0;m=couldn’t execute POST against https://gitlab.com/api/v4/jobs/request: Post https://gitlab.com/api/v4/jobs/request: read tcp 172.20.103.24:35902->35.231.145.151:443: read: connection timed out

What does it mean ?

2

Well, we are trying (with admin) to decipher why the connection got timed out, but still no way…

3

Hi @miroi and welcome to the community forum!

Given the error messages you provided, it appears that there’s a network issue preventing <your-ip:35902> from connecting to <gitlab-ip:443>.

There are a few commands we can use to confirm this is true by running them on the runner instance:

  • try to ping GitLab.com by running
    ping gitlab.com
  • can you connect to the gitlab.com api using curl?
    curl "https://gitlab.com/api/v4/projects/250833"

If either of these commands fails to connect, you can use a utility like traceroute (ex: traceroute gitlab.com) to indicate where the connection is failing. I’m not familiar with your firewall setup, but I suspect the connection is being terminiated at the network level on your side.

1 Like
4

Hi Greg !

We tried all commands, they are working. We continue to examine what could be wrong.

[root@adf2 ~]#  ping gitlab.com
PING gitlab.com (35.231.145.151) 56(84) bytes of data.
64 bytes from 151.145.231.35.bc.googleusercontent.com (35.231.145.151): icmp_seq=1 ttl=40 time=114 ms
64 bytes from 151.145.231.35.bc.googleusercontent.com (35.231.145.151): icmp_seq=2 ttl=40 time=114 ms
^C
--- gitlab.com ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 114.626/114.630/114.634/0.004 
ms
[root@adf2 ~]#  curl "https://gitlab.com/api/v4/projects/250833"
{"id":250833,"description":"GitLab Runner is the open source project that is used to run your CI/CD jobs and send the results back to GitLab","name":"gitlab-runner","name_with_namespace":"GitLab.org / gitlab-runner","path":"gitlab-runner","path_with_namespace":"gitlab-org/gitlab-runner","created_at":"2015-04-27T21:10:25.322Z","default_branch":"master","tag_list":[],"ssh_url_to_repo":"git@gitlab.com:gitlab-org/gitlab-runner.git","http_url_to_repo":"https://gitlab.com/gitlab-org/gitlab-runner.git","web_url":"https://gitlab.com/gitlab-org/gitlab-runner","readme_url":"https://gitlab.com/gitlab-org/gitlab-runner/blob/master/README.md","avatar_url":"https://assets.gitlab-static.net/uploads/-/system/project/avatar/250833/runner_logo.png","star_count":1386,"forks_count":1075,"last_activity_at":"2019-08-16T07:51:46.750Z","namespace":{"id":9970,"name":"GitLab.org","path":"gitlab-org","kind":"group","full_path":"gitlab-org","parent_id":null,"avatar_url":"/uploads/-/system/group/avatar/9970/logo-extra-whitespace.png","web_url":"https://gitlab.com/groups/gitlab-org"}}[root@adf2 ~]# 

[root@adf2 ~]# traceroute gitlab.com
traceroute to gitlab.com (35.231.145.151), 30 hops max, 60 byte packets
 1  gateway (172.16.0.1)  0.138 ms  0.111 ms  0.223 ms
 2  194.160.44.225 (194.160.44.225)  0.527 ms  0.475 ms  0.465 ms
 3  UMB-Banska-Bystrica.sanet2.sk (194.160.185.252)  0.630 ms  0.588 ms  0.698 ms
 4  CVT-Bratislava.sanet2.sk (194.160.8.1)  3.363 ms  3.353 ms  3.535 ms
 5  r98-bm.cesnet.cz (195.113.179.165)  8.637 ms  8.996 ms  8.601 ms
 6  195.113.235.109 (195.113.235.109)  15.156 ms  13.267 ms  13.240 ms
 7  r2-r93.cesnet.cz (195.113.157.70)  10.581 ms  15.700 ms  15.681 ms
 8  108.170.245.50 (108.170.245.50)  10.708 ms 108.170.245.34 (108.170.245.34)  10.621 ms  16.973 ms
 9  108.170.236.69 (108.170.236.69)  19.226 ms 209.85.142.253 (209.85.142.253)  23.439 ms  23.420 ms
10  72.14.239.245 (72.14.239.245)  18.578 ms 108.170.228.9 (108.170.228.9)  18.980 ms 209.85.245.30 (209.85.245.30)  45.901 ms
11  108.170.236.120 (108.170.236.120)  29.435 ms  26.009 ms  24.725 ms
12  108.170.234.119 (108.170.234.119)  30.549 ms 108.170.237.243 (108.170.237.243)  30.349 ms 108.170.234.119 (108.170.234.119)  30.251 ms
13  172.253.51.230 (172.253.51.230)  104.416 ms 72.14.239.190 (72.14.239.190)  104.087 ms  104.066 ms
14  216.239.40.20 (216.239.40.20)  114.938 ms  115.269 ms 209.85.255.252 (209.85.255.252)  115.813 ms
15  216.239.41.240 (216.239.41.240)  117.808 ms 216.239.35.187 (216.239.35.187)  116.689 ms 216.239.50.204 (216.239.50.204)  117.998 ms
16  * 172.253.66.169 (172.253.66.169)  116.175 ms 172.253.66.167 (172.253.66.167)  116.666 ms
17  * * *
18  * * *
19  * * *
20  * * *
21  * * *
22  * * *
23  * * *
24  * * *
25  151.145.231.35.bc.googleusercontent.com (35.231.145.151)  115.931 ms  115.922 ms *
[root@adf2 ~]#
5

Well,

while checking logs I found this below. I guess "Checking for jobs… failed " means interrupted connection to gitlab.com ?

[root@adf2 ~]#  journalctl -u gitlab-runner
-- Logs begin at Mon 2019-08-05 13:02:19 CEST, end at Fri 2019-08-16 11:19:29 CEST. --
Aug 05 13:02:38 adf2 systemd[1]: Started GitLab Runner.
-- Logs begin at Mon 2019-08-05 13:02:19 CEST, end at Fri 2019-08-16 11:19:29 CEST. --
Aug 05 13:02:38 adf2 systemd[1]: Started GitLab Runner.
Aug 05 13:02:40 adf2 gitlab-runner[6464]: Runtime platform                                    arch=amd64 os=linux pid=6464 revision=ac2a293
Aug 05 13:02:40 adf2 gitlab-runner[6464]: Starting multi-runner from /etc/gitlab-runner/config.toml ...  builds=0
Aug 05 13:02:40 adf2 gitlab-runner[6464]: Starting multi-runner from /etc/gitlab-runner/config.toml ...  builds=0
Aug 05 13:02:40 adf2 gitlab-runner[6464]: Running in system-mode.                           
Aug 05 13:02:40 adf2 gitlab-runner[6464]: 
Aug 05 13:02:40 adf2 gitlab-runner[6464]: Running in system-mode.                           
Aug 05 13:02:40 adf2 gitlab-runner[6464]:                                                   
Aug 05 13:02:40 adf2 gitlab-runner[6464]: Configuration loaded                                builds=0
Aug 05 13:02:40 adf2 gitlab-runner[6464]: Configuration loaded                                builds=0
Aug 05 13:02:40 adf2 gitlab-runner[6464]: listen_address not defined, metrics & debug endpoints disabled  builds=0
Aug 05 13:02:40 adf2 gitlab-runner[6464]: listen_address not defined, metrics & debug endpoints disabled  builds=0
Aug 05 13:02:40 adf2 gitlab-runner[6464]: [session_server].listen_address not defined, session endpoints disabled  builds=0
Aug 05 13:02:40 adf2 gitlab-runner[6464]: [session_server].listen_address not defined, session endpoints disabled  builds=0
Aug 14 11:46:34 adf2 gitlab-runner[6464]: WARNING: Checking for jobs... failed                runner=0472a85c status=503 Service Unavailabl
Aug 14 11:46:34 adf2 gitlab-runner[6464]: WARNING: Checking for jobs... failed                runner=0472a85c status=503 Service Unavailabl
Aug 14 11:47:04 adf2 gitlab-runner[6464]: WARNING: Checking for jobs... failed                runner=0472a85c status=503 Service Unavailabl
Aug 14 11:47:04 adf2 gitlab-runner[6464]: WARNING: Checking for jobs... failed                runner=0472a85c status=503 Service Unavailabl
Aug 14 11:47:35 adf2 gitlab-runner[6464]: WARNING: Checking for jobs... failed                runner=0472a85c status=503 Service Unavailabl
Aug 14 11:47:35 adf2 gitlab-runner[6464]: WARNING: Checking for jobs... failed                runner=0472a85c status=503 Service Unavailabl
Aug 14 11:48:05 adf2 gitlab-runner[6464]: WARNING: Checking for jobs... failed                runner=0472a85c status=503 Service Unavailabl
Aug 14 11:48:05 adf2 gitlab-runner[6464]: WARNING: Checking for jobs... failed                runner=0472a85c status=503 Service Unavailabl
Aug 14 11:48:36 adf2 gitlab-runner[6464]: WARNING: Checking for jobs... failed                runner=0472a85c status=503 Service Unavailabl
Aug 14 11:48:36 adf2 gitlab-runner[6464]: WARNING: Checking for jobs... failed                runner=0472a85c status=503 Service Unavailabl
Aug 14 11:49:09 adf2 gitlab-runner[6464]: WARNING: Checking for jobs... failed                runner=0472a85c status=503 Service Unavailabl
Aug 14 11:49:09 adf2 gitlab-runner[6464]: WARNING: Checking for jobs... failed                runner=0472a85c status=503 Service Unavailabl
Aug 14 11:49:41 adf2 gitlab-runner[6464]: WARNING: Checking for jobs... failed                runner=0472a85c status=503 Service Unavailabl
Aug 14 11:49:41 adf2 gitlab-runner[6464]: WARNING: Checking for jobs... failed                runner=0472a85c status=503 Service Unavailabl
Aug 14 11:50:22 adf2 gitlab-runner[6464]: WARNING: Checking for jobs... failed                runner=0472a85c status=503 Service Unavailabl
Aug 14 11:50:22 adf2 gitlab-runner[6464]: WARNING: Checking for jobs... failed                runner=0472a85c status=503 Service Unavailabl
Aug 14 11:50:53 adf2 gitlab-runner[6464]: WARNING: Checking for jobs... failed                runner=0472a85c status=503 Service Unavailabl
Aug 14 11:50:53 adf2 gitlab-runner[6464]: WARNING: Checking for jobs... failed                runner=0472a85c status=503 Service Unavailabl
Aug 14 11:52:13 adf2 gitlab-runner[6464]: WARNING: Checking for jobs... failed                runner=0472a85c status=503 Service Unavailabl
Aug 14 11:52:13 adf2 gitlab-runner[6464]: WARNING: Checking for jobs... failed                runner=0472a85c status=503 Service Unavailabl
Aug 14 11:53:43 adf2 gitlab-runner[6464]: WARNING: Checking for jobs... failed                runner=0472a85c status=503 Service Unavailabl
Aug 14 11:53:43 adf2 gitlab-runner[6464]: WARNING: Checking for jobs... failed                runner=0472a85c status=503 Service Unavailabl
Aug 14 11:54:37 adf2 gitlab-runner[6464]: WARNING: Checking for jobs... failed                runner=0472a85c status=503 Service Unavailabl
Aug 14 11:54:37 adf2 gitlab-runner[6464]: WARNING: Checking for jobs... failed                runner=0472a85c status=503 Service Unavailabl
Aug 14 11:56:07 adf2 gitlab-runner[6464]: WARNING: Checking for jobs... failed                runner=0472a85c status=503 Service Unavailabl
Aug 14 11:56:07 adf2 gitlab-runner[6464]: WARNING: Checking for jobs... failed                runner=0472a85c status=503 Service Unavailabl
Aug 14 12:57:22 adf2 gitlab-runner[6464]: WARNING: Checking for jobs... failed                runner=0472a85c status=503 Service Unavailabl
Aug 14 12:57:22 adf2 gitlab-runner[6464]: WARNING: Checking for jobs... failed                runner=0472a85c status=503 Service Unavailabl
Aug 14 12:57:53 adf2 gitlab-runner[6464]: WARNING: Checking for jobs... failed                runner=0472a85c status=503 Service Unavailabl
Aug 14 12:57:53 adf2 gitlab-runner[6464]: WARNING: Checking for jobs... failed                runner=0472a85c status=503 Service Unavailabl
Aug 14 12:58:23 adf2 gitlab-runner[6464]: WARNING: Checking for jobs... failed                runner=0472a85c status=503 Service Unavailabl
Aug 14 12:58:23 adf2 gitlab-runner[6464]: WARNING: Checking for jobs... failed                runner=0472a85c status=503 Service Unavailabl
Aug 14 12:58:53 adf2 gitlab-runner[6464]: WARNING: Checking for jobs... failed                runner=0472a85c status=503 Service Unavailabl
Aug 14 12:58:53 adf2 gitlab-runner[6464]: WARNING: Checking for jobs... failed                runner=0472a85c status=503 Service Unavailabl
Aug 14 12:59:24 adf2 gitlab-runner[6464]: WARNING: Checking for jobs... failed                runner=0472a85c status=503 Service Unavailabl
Aug 14 12:59:24 adf2 gitlab-runner[6464]: WARNING: Checking for jobs... failed                runner=0472a85c status=503 Service Unavailabl
Aug 14 14:00:39 adf2 gitlab-runner[6464]: WARNING: Checking for jobs... failed                runner=0472a85c status=503 Service Unavailabl
Aug 14 14:00:39 adf2 gitlab-runner[6464]: WARNING: Checking for jobs... failed                runner=0472a85c status=503 Service Unavailabl
[root@adf2 ~]#
6

Ah, our admin find this explanation : https://gitlab.com/gitlab-com/support-forum/issues/3322
Could you please comment on that ?

7

Have you been able to solve it @miroi ? Support forum with issues is now closed