Gitlab webhook URL not working on https SSL

Hi ,

I’ve my Mattermost setup in Google Cloud Platform. When I tried to test my webhook connection from Gitlab (local server), there is some error message like below, captured from “gitlab-ctl tail” command:

OpenSSL::SSL::SSLError (SSL_connect returned=1 errno=0 state=error: certificate verify failed):
app/models/hooks/web_hook.rb:39:in execute' app/models/hooks/service_hook.rb:23:in execute’
app/models/project_services/jenkins_service.rb:20:in execute' app/models/service.rb:96:in test’
app/controllers/projects/services_controller.rb:43:in `test’

I’m using Letsencrypt for my Mattermost setup, Mattermost seems working fine but the webhook Notification just not working.

Is it something wrong with my SSL Certificate? I’ve no idea on how to troubleshoot this issue as I’m new to Gitlab and Mattermost.

Please help, very much appreciated.

What’s reported if you do an openssl s_client -connect $hostname:443 ?

Hi @jonathon,

root@3LC-GitLab2:~# openssl s_client -connect $hostname:443
gethostbyname failure

Hi @jonathon,

I got this:

root@3LC-GitLab2:~# openssl s_client -connect localhost:443
depth=1 C = US, O = Let’s Encrypt, CN = Let’s Encrypt Authority X3
verify error:num=20:unable to get local issuer certificate
verify return:0

Certificate chain
0 s:/
i:/C=US/O=Let’s Encrypt/CN=Let’s Encrypt Authority X3
1 s:/C=US/O=Let’s Encrypt/CN=Let’s Encrypt Authority X3
i:/O=Digital Signature Trust Co./CN=DST Root CA X3

Server certificate
issuer=/C=US/O=Let’s Encrypt/CN=Let’s Encrypt Authority X3

No client certificate CA names sent

SSL handshake has read 3129 bytes and written 421 bytes

New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
Protocol : TLSv1.2
Cipher : ECDHE-RSA-AES256-GCM-SHA384
Session-ID: 958950D8C56FD5D34685791A20628AC052AF7D36844BE4464A65C256E827CC02
Master-Key: 443E3FD6B5B6932C35C93B4A7302B23EF6C7B8E12EB34DE3248D30AFFDA8D05ECE7EE51EEF8CE4F4849B8E9F9E5FDD3A
Key-Arg : None
PSK identity: None
PSK identity hint: None
SRP username: None
TLS session ticket lifetime hint: 300 (seconds)
TLS session ticket:
0000 - 2d cb 92 eb 95 bb 89 42-e6 ef b6 ee 88 d4 15 7a -…B…z
0010 - e9 5a 03 15 91 6c 10 15-9c 0f 08 95 1f 29 74 4a .Z…l…)tJ
0020 - 90 ad 1f 24 b9 d1 30 5f-0b 7f 82 79 33 74 51 14 …$…0_…y3tQ.
0030 - 78 d1 82 f4 58 73 77 12-7c f4 15 af 06 06 b6 98 x…Xsw.|…
0040 - 34 3c 96 c5 dc 50 4f 07-ea ca 9a c4 32 fb 0f 73 4<…PO…2…s
0050 - af 15 f3 f0 0c 11 40 27-09 93 25 b2 e3 2d 3b 3f …@’…%…-;?
0060 - 0f e6 5f 5a 4d f9 5e b4-6b 43 0a 4a 47 df ad 71 …_ZM.^.kC.JG…q
0070 - 39 79 5e f6 f4 aa fc 32-48 9c 4f 88 16 ff d3 40 9y^…2H.O…@
0080 - 61 ca 1f a4 4c b8 56 91-19 64 44 df 44 20 68 f6 a…L.V…dD.D h.
0090 - 82 4e 3b 7c 78 fb 3a eb-48 c0 f7 4d 99 93 ed 75 .N;|x.:.H…M…u
00a0 - 31 4a 46 94 22 c1 64 65-10 fe cc 8a 17 50 4e 62 1JF.".de…PNb

Start Time: 1504268567
Timeout   : 300 (sec)
Verify return code: 20 (unable to get local issuer certificate)


Have you set up localhost as an SSL-enabled server? Would you not be better trying to connect to its FQDN?

Hi @jonathon,
My Gitlab is hosted locally in my local network with http://192.168.x.x hence I don’t have a FQDN for it.

Let’s Encrypt can’t provide an SSL certificate for an internal IP address.

It’s also not going to work if you’ve configured to use their CA certificate along with a local self-signed cert.

Hi @jonathon,
Thanks for the reply. Does it mean that The only way to make it work is by having a FQDN for my gitlab server so that the webhook can work over https?

If you’re wanting to use Let’s Encrypt, yes. Otherwise, it might work with a (correctly configured) self-signed certificate.

Hi @jonathon,

Just to clarify my scenario.

  1. I have Mattermost setup in Google Cloud,, with Letsencrypt installed

  2. I have GitLab installed in my local network http://192.168.x.x

  3. I’ve enabled “Incoming Webhook” in Mattermost, and generated a webhook URL

  4. Login to Gitlab > Project > Project NAme > Settings Icon > Services > Slack, and paste the URL to the “Webhook” column, and click on “Test settings”.

Ok now the problem is when I click on the “Test settings” button, it will prompt “500, Whoops, something went wrong on our end”, and I login to Gitlab server end to check the log using “gitlab-ctl tail”, and I see the below message:

==> /var/log/gitlab/gitlab-rails/production.log <==
Started GET “/3LCsystems/iOS/services/slack/test” for at 2017-09-05 11:02:45 +0800
Processing by Projects::ServicesController#test as HTML
Parameters: {“namespace_id”=>“3LCsystems”, “project_id”=>“iOS”, “id”=>“slack”}
Completed 500 Internal Server Error in 160ms (ActiveRecord: 7.9ms)

OpenSSL::SSL::SSLError (SSL_connect returned=1 errno=0 state=error: certificate verify failed):
app/models/project_services/slack_service.rb:79:in execute' app/models/service.rb:116:in test’
app/controllers/projects/services_controller.rb:34:in test' lib/gitlab/request_profiler/middleware.rb:15:in call’
lib/gitlab/middleware/go.rb:16:in `call’

I’ve spent 2 days googling for the solutions but no luck. I’ve tried this on installing CA whereby the certs are belongs to my domain from Letsencrypt.

I’ve tested using the following command in my gitlab server, and also on my own macbook terminal and the webhook seems working fine:

curl -i -X POST -d ‘payload={“text”: “Hello, this is some text\nThis is more text. :tada:”}’

But still, the same error message prompting when I use the https webhook URL link in Gitlab. Scratching my head for few days because of this issue, now my gitlab webhook is unable to work in Mattermost Google .

Any suggestion on this issue? Million of thanks.