Gitlab webhook URL not working on https SSL

How to Use GitLab
1

Hi ,

I’ve my Mattermost setup in Google Cloud Platform. When I tried to test my webhook connection from Gitlab (local server), there is some error message like below, captured from “gitlab-ctl tail” command:

===================================================
OpenSSL::SSL::SSLError (SSL_connect returned=1 errno=0 state=error: certificate verify failed):
app/models/hooks/web_hook.rb:39:in execute' app/models/hooks/service_hook.rb:23:in execute’
app/models/project_services/jenkins_service.rb:20:in execute' app/models/service.rb:96:in test’
app/controllers/projects/services_controller.rb:43:in `test’

I’m using Letsencrypt for my Mattermost setup, Mattermost seems working fine but the webhook Notification just not working.

Is it something wrong with my SSL Certificate? I’ve no idea on how to troubleshoot this issue as I’m new to Gitlab and Mattermost.

Please help, very much appreciated.

2

What’s reported if you do an openssl s_client -connect $hostname:443 ?

3

Hi @jonathon,

root@3LC-GitLab2:~# openssl s_client -connect $hostname:443
gethostbyname failure
connect:errno=0

4

Hi @jonathon,

I got this:

==========================================================
root@3LC-GitLab2:~# openssl s_client -connect localhost:443
CONNECTED(00000003)
depth=1 C = US, O = Let’s Encrypt, CN = Let’s Encrypt Authority X3
verify error:num=20:unable to get local issuer certificate
verify return:0

Certificate chain
0 s:/CN=chat.3lc.my
i:/C=US/O=Let’s Encrypt/CN=Let’s Encrypt Authority X3
1 s:/C=US/O=Let’s Encrypt/CN=Let’s Encrypt Authority X3
i:/O=Digital Signature Trust Co./CN=DST Root CA X3

Server certificate
-----BEGIN CERTIFICATE-----
MIIE+TCCA+GgAwIBAgISBJrAwf9m8Br44b6CKyXEAUzTMA0GCSqGSIb3DQEBCwUA
MEoxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MSMwIQYDVQQD
ExpMZXQncyBFbmNyeXB0IEF1dGhvcml0eSBYMzAeFw0xNzA4MjkxMDA3MDBaFw0x
NzExMjcxMDA3MDBaMBYxFDASBgNVBAMTC2NoYXQuM2xjLm15MIIBIjANBgkqhkiG
9w0BAQEFAAOCAQ8AMIIBCgKCAQEAubATGEnaxLNsl7ULqqn+xwRkV/4cs31V8yts
nGFsS2JUi59I55cqp562AmfEaycn7CUfnfTmoMj5Mcfb/C8MLBAj9jUL2Ckq4HJb
sBlnMMV38nUbN5Q7YcTxaYkT9tSKDBVq9GGd+hjWO1/YXGXmsY7mp8U/jnJPNtkT
498lBM11Ob95T34c+dSISvsnrfVXn9BXeH+f97MrjuzWyzedrahTh8Utckyeo5/8
2epm6T+td8Idw2gaZTMjFOsrLZZn/ifHAUTSkB7HStfj9HeJvzlG7oD89hC6GvNJ
FHIMXw2Cd/xi/vBHpYjWjJ3AedwW1oO4ugy24wnJZKUtNBkDMQIDAQABo4ICCzCC
AgcwDgYDVR0PAQH/BAQDAgWgMB0GA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcD
AjAMBgNVHRMBAf8EAjAAMB0GA1UdDgQWBBQmjCLiBkOwbpmsf77CZPIrvq+arTAf
BgNVHSMEGDAWgBSoSmpjBH3duubRObemRWXv86jsoTBvBggrBgEFBQcBAQRjMGEw
LgYIKwYBBQUHMAGGImh0dHA6Ly9vY3NwLmludC14My5sZXRzZW5jcnlwdC5vcmcw
LwYIKwYBBQUHMAKGI2h0dHA6Ly9jZXJ0LmludC14My5sZXRzZW5jcnlwdC5vcmcv
MBYGA1UdEQQPMA2CC2NoYXQuM2xjLm15MIH+BgNVHSAEgfYwgfMwCAYGZ4EMAQIB
MIHmBgsrBgEEAYLfEwEBATCB1jAmBggrBgEFBQcCARYaaHR0cDovL2Nwcy5sZXRz
ZW5jcnlwdC5vcmcwgasGCCsGAQUFBwICMIGeDIGbVGhpcyBDZXJ0aWZpY2F0ZSBt
YXkgb25seSBiZSByZWxpZWQgdXBvbiBieSBSZWx5aW5nIFBhcnRpZXMgYW5kIG9u
bHkgaW4gYWNjb3JkYW5jZSB3aXRoIHRoZSBDZXJ0aWZpY2F0ZSBQb2xpY3kgZm91
bmQgYXQgaHR0cHM6Ly9sZXRzZW5jcnlwdC5vcmcvcmVwb3NpdG9yeS8wDQYJKoZI
hvcNAQELBQADggEBAE3iuaFp+Qo/pr91e/o6lsaIq46U2XE9e69EOQV6IlTAhPoZ
wSNAVIcSEcZLHVkPixnQgPdSOVC7bSM1tAxdh5zp5k86JM0qkdZ/xVTjKlYlpt7H
nhkwYY27d86Hz0FGi3uA6bahfUY1DFApOd/N/9VAZWcRsU90wOBaysp0fX4tFV5R
A2Gm5GzqmfOeVl4P29vMISzos5VlzpPPUaJythf9M6aijRGd9QYysoyHFIC825lC
gcU8M8xtZIFg+zQVe1hua11vL78duPdg4EA+3kpbhfXP7icdCxxIvYpuoM5nU3dJ
syKd/0QqZS+Xdxqttl2nZV8ecBMEGYfLMUziU2w=
-----END CERTIFICATE-----
subject=/CN=chat.3lc.my
issuer=/C=US/O=Let’s Encrypt/CN=Let’s Encrypt Authority X3

No client certificate CA names sent

SSL handshake has read 3129 bytes and written 421 bytes

New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
Protocol : TLSv1.2
Cipher : ECDHE-RSA-AES256-GCM-SHA384
Session-ID: 958950D8C56FD5D34685791A20628AC052AF7D36844BE4464A65C256E827CC02
Session-ID-ctx:
Master-Key: 443E3FD6B5B6932C35C93B4A7302B23EF6C7B8E12EB34DE3248D30AFFDA8D05ECE7EE51EEF8CE4F4849B8E9F9E5FDD3A
Key-Arg : None
PSK identity: None
PSK identity hint: None
SRP username: None
TLS session ticket lifetime hint: 300 (seconds)
TLS session ticket:
0000 - 2d cb 92 eb 95 bb 89 42-e6 ef b6 ee 88 d4 15 7a -…B…z
0010 - e9 5a 03 15 91 6c 10 15-9c 0f 08 95 1f 29 74 4a .Z…l…)tJ
0020 - 90 ad 1f 24 b9 d1 30 5f-0b 7f 82 79 33 74 51 14 …$…0_…y3tQ.
0030 - 78 d1 82 f4 58 73 77 12-7c f4 15 af 06 06 b6 98 x…Xsw.|…
0040 - 34 3c 96 c5 dc 50 4f 07-ea ca 9a c4 32 fb 0f 73 4<…PO…2…s
0050 - af 15 f3 f0 0c 11 40 27-09 93 25 b2 e3 2d 3b 3f …@’…%…-;?
0060 - 0f e6 5f 5a 4d f9 5e b4-6b 43 0a 4a 47 df ad 71 …_ZM.^.kC.JG…q
0070 - 39 79 5e f6 f4 aa fc 32-48 9c 4f 88 16 ff d3 40 9y^…2H.O…@
0080 - 61 ca 1f a4 4c b8 56 91-19 64 44 df 44 20 68 f6 a…L.V…dD.D h.
0090 - 82 4e 3b 7c 78 fb 3a eb-48 c0 f7 4d 99 93 ed 75 .N;|x.:.H…M…u
00a0 - 31 4a 46 94 22 c1 64 65-10 fe cc 8a 17 50 4e 62 1JF.".de…PNb

Start Time: 1504268567
Timeout   : 300 (sec)
Verify return code: 20 (unable to get local issuer certificate)

====================================================

5

Have you set up localhost as an SSL-enabled server? Would you not be better trying to connect to its FQDN?

6

Hi @jonathon,
My Gitlab is hosted locally in my local network with http://192.168.x.x hence I don’t have a FQDN for it.

7

Let’s Encrypt can’t provide an SSL certificate for an internal IP address.

It’s also not going to work if you’ve configured to use their CA certificate along with a local self-signed cert.

8

Hi @jonathon,
Thanks for the reply. Does it mean that The only way to make it work is by having a FQDN for my gitlab server so that the webhook can work over https?

9

If you’re wanting to use Let’s Encrypt, yes. Otherwise, it might work with a (correctly configured) self-signed certificate.

10

Hi @jonathon,

Just to clarify my scenario.

  1. I have Mattermost setup in Google Cloud, https://chat.3lc.my:8065, with Letsencrypt installed

  2. I have GitLab installed in my local network http://192.168.x.x

  3. I’ve enabled “Incoming Webhook” in Mattermost, and generated a webhook URL https://chat.3lc.my:8065/hooks/xxxxxx

  4. Login to Gitlab > Project > Project NAme > Settings Icon > Services > Slack, and paste the URL to the “Webhook” column, and click on “Test settings”.

Ok now the problem is when I click on the “Test settings” button, it will prompt “500, Whoops, something went wrong on our end”, and I login to Gitlab server end to check the log using “gitlab-ctl tail”, and I see the below message:

==============================================
==> /var/log/gitlab/gitlab-rails/production.log <==
Started GET “/3LCsystems/iOS/services/slack/test” for 192.168.2.240 at 2017-09-05 11:02:45 +0800
Processing by Projects::ServicesController#test as HTML
Parameters: {“namespace_id”=>“3LCsystems”, “project_id”=>“iOS”, “id”=>“slack”}
Completed 500 Internal Server Error in 160ms (ActiveRecord: 7.9ms)

OpenSSL::SSL::SSLError (SSL_connect returned=1 errno=0 state=error: certificate verify failed):
app/models/project_services/slack_service.rb:79:in execute' app/models/service.rb:116:in test’
app/controllers/projects/services_controller.rb:34:in test' lib/gitlab/request_profiler/middleware.rb:15:in call’
lib/gitlab/middleware/go.rb:16:in `call’

I’ve spent 2 days googling for the solutions but no luck. I’ve tried this on installing CA whereby the certs are belongs to my chat.3lc.my domain from Letsencrypt.

I’ve tested using the following command in my gitlab server, and also on my own macbook terminal and the webhook seems working fine:

curl -i -X POST -d ‘payload={“text”: “Hello, this is some text\nThis is more text. :tada:”}’ https://chat.3lc.my:8065/hooks/xxxxxx

But still, the same error message prompting when I use the https webhook URL link in Gitlab. Scratching my head for few days because of this issue, now my gitlab webhook is unable to work in Mattermost Google .

Any suggestion on this issue? Million of thanks.