Websocket error

How to Use GitLab Self-managed
1

Hi,

I recently noticed an error on my self-hosted instance when viewing merge requests:

WebSocket: connect to …/-cable failed

as also shown in

this screenshot
this screenshot712×278 44.8 KB
. The error seems to prevent merging, since possible actions never get loaded. I suspect a proxy configuration error (I proxy through apache); but I’m not quite sure how to debug this one.

Has anyone any hint on this?

Thanks!

2

I did a bit of investiation of the issue. It really seems to be a proxying error. The main proxy configuration looks like this:

ProxyAddHeaders On
RequestHeader add X-Forwarded-Ssl on
RequestHeader set X-Forwarded-Proto "https"

ProxyPass unix:///opt/gitlab/gitlab/tmp/sockets/gitlab-workhorse.socket|
http://127.0.0.1/
ProxyPassReverse unix:///opt/gitlab/gitlab/tmp/sockets/gitlab-
workhorse.socket|http://127.0.0.1/

which results in

Started GET "/-/cable" for $REMOTE_IP at 2022-12-22 14:35:51 +0100
Started GET "/-/cable/"[non-WebSocket] for $REMOTE_IP at 2022-12-22 14:35:51 
+0100
Failed to upgrade to WebSocket (REQUEST_METHOD: GET, HTTP_CONNECTION: , 
HTTP_UPGRADE: )
Finished "/-/cable/"[non-WebSocket] for $REMOTE_IP at 2022-12-22 14:35:51 
+0100

Using Apache’s wstunnel module, I need to add some rewriting (below the only working version of rewriting to a UNIX socket, I found to be at least proxing):

RewriteEngine on
RewriteCond %{HTTP:Upgrade} websocket [NC]
RewriteCond %{HTTP:Connection} upgrade [NC]
RewriteRule ^/?(.*) "unix:/opt/gitlab/gitlab/tmp/sockets/gitlab-
workhorse.socket|http://127.0.0.1/$1" [P,NE]

This results in:

Started GET "/proxy:http://127.0.0.1/-/cable/" for $REMOTE_IP at 2022-12-22 
14:46:19 +0100
Processing by ApplicationController#route_not_found as HTML
  Parameters: {"unmatched_route"=>"proxy:http:/127.0.0.1/-/cable"}
  Rendered layout layouts/errors.html.haml (Duration: 2.2ms | Allocations: 
600)
Completed 404 Not Found in 30ms (Views: 2.8ms | ActiveRecord: 3.5ms | 
Elasticsearch: 0.0ms | Allocations: 7303)

I’m not really sure how to dig further into the problem; the error is surely within the RewriteRule, there should be a ws://127.0.0.1/$1 instead of the http; but this results in a configuration error within Apache…

3

Found a version having the http upgrade headers:

RequestHeader add X-Forwarded-Ssl on
RequestHeader set X-Forwarded-Proto "https"

<Proxy *>
    Require all granted
</Proxy>

<Location />
    ProxyPass unix:///opt/gitlab/gitlab/tmp/sockets/gitlab-workhorse.socket|http://127.0.0.1/
    ProxyPassReverse unix:///opt/gitlab/gitlab/tmp/sockets/gitlab-workhorse.socket|http://127.0.0.1/ </Location>

<Location /-/cable>
    ProxyPass unix:///opt/gitlab/gitlab/tmp/sockets/gitlab-workhorse.socket|ws://127.0.0.1/-/cable
    ProxyPassReverse unix:///opt/gitlab/gitlab/tmp/sockets/gitlab-workhorse.socket|ws://127.0.0.1/-/cable
</Location>

making the logs look like that:

Started GET "/-/cable" for $REMOTE_IP at 2022-12-23 16:02:43 +0100
Started GET "/-/cable/" [WebSocket] for $REMOTE_IP at 2022-12-23 16:02:43 +0100
Request origin not allowed: https://git.jan-kohnert.de
Failed to upgrade to WebSocket (REQUEST_METHOD: GET, HTTP_CONNECTION: Upgrade, HTTP_UPGRADE: websocket)
Finished "/-/cable/" [WebSocket] for $REMOTE_IP at 2022-12-23 16:02:43 +0100

https://git.jan-kohnert.de is my server, so I think this should be fine. However, I’m probably missing a GitLab config directive. Unforftunately GitLab docs don’t tell any configuration details.

4

Since the tunneling itself works; I’m closing this one, and create a new post with a better matching subject.

5

Having the same issue but I couldn’t get it to work with the examples above. Raised an issue at How to configure apache proxy to also support websockets for /-/cable (#88) · Issues · GitLab.org / GitLab recipes · GitLab hoping for some feedback there.

6

Have you tried the solution I posted in my other thread redarding this topic? The solution presented here is missing one crutial configuration option: ProxyPreserveHost on is needed to make the whole thing work correctly. Also, this <Proxy *>… should not be needed, but the configuration works regardless of this setting.

7

Yes, I have had the ProxyPreserveHost on setting already, but not the <Proxy *> setting. So I added that.

Also, I’ve had the request header settings too, however they had been using mixed hyphen and underscores. I’ve fixed that too and defined both variables with both hyphen and underscore, just be be save :wink:

Not seeing cable errors anymore, so one of that must have fixed it.