Tafka
January 6, 2023, 8:42am
1
Hi
I have a question regarding the /-/cable endpoint.
We are running a selfhosted CE gitlab instance and our monitoring started to notice that users with vscode gitlab extension got a lot of 404 to the /-/cable endpoint.
As I understand this is the ActionCable util from Puma in Ruby.
Somebody has written about it here, but he uses a instance that does not run in docker.
I did a bit of investiation of the issue. It really seems to be a proxying error. The main proxy configuration looks like this:
ProxyAddHeaders On
RequestHeader add X-Forwarded-Ssl on
RequestHeader set X-Forwarded-Proto "https"
ProxyPass unix:///opt/gitlab/gitlab/tmp/sockets/gitlab-workhorse.socket|
http://127.0.0.1/
ProxyPassReverse unix:///opt/gitlab/gitlab/tmp/sockets/gitlab-
workhorse.socket|http://127.0.0.1/
which results in
Started GET "/-/cable" for $REMOTE_IP at 2022-12-22 14:35:51 +…
The docs only specify amount of workers.
I found this website that exposes all the usable ports.
For Puma it is 8080, because ActionCable is puma, do I need to use port 8080 when forwarding front proxy messages to the container?
my conf:
# ports from docker ps command
0.0.0.0:22->22/tcp, :::22->22/tcp, 0.0.0.0:80->80/tcp, :::80->80/tcp, 443/tcp, 127.0.0.1:8060->8060/tcp, 127.0.0.1:5051->5050/tcp
# nginx conf
server {
listen 80;
server_name REDACTED;
return 301 REDACTED$request_uri;
}
server {
listen 443 ssl http2;
server_name REDACTED;
index index.html index.htm;
client_max_body_size 0;
ssl_certificate /etc/ssl/certs/REDACTED.pem;
ssl_certificate_key /etc/ssl/private/REDACTED.pem;
location / {
proxy_pass http://127.0.0.1:80;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto "https";
}
location /-/kubernetes-agent/ {
proxy_set_header Host $host;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_pass http://127.0.0.1:80;
}
location /-/cable {
proxy_set_header Host $host;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_pass http://127.0.0.1:80;
}
}
EDIT:
Also added one line from /var/log/nginx/access.log
REDACTED.IP - - [06/Jan/2023:12:32:44 +0200] "GET /-/cable HTTP/1.1" 404 1640 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36" "-"
Tafka
January 9, 2023, 9:22am
2
I tried to use this Nginx configuration that was present here, but to no avail:
Tafka
January 9, 2023, 9:31am
3
Started GET "/-/cable" for 172.17.0.1 at 2023-01-09 09:29:38 +0000
Started GET "/-/cable/"[non-WebSocket] for 172.17.0.1 at 2023-01-09 09:29:38 +0000
Finished "/-/cable/"[non-WebSocket] for 172.17.0.1 at 2023-01-09 09:29:38 +0000
{"content_type":"text/html; charset=utf-8","correlation_id":"01GPAXTBCEE6QW84QMVDH1MV3F","duration_ms":40,"host":"REDACTED","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"127.0.0.1:0","remote_ip":"127.0.0.1","route":"^/-/","status":404,"system":"http","time":"2023-01-09T09:29:38Z","ttfb_ms":39,"uri":"/-/cable","user_agent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:108.0) Gecko/20100101 Firefox/108.0","written_bytes":3207}
172.17.0.1 - - [09/Jan/2023:09:29:38 +0000] "GET /-/cable HTTP/1.0" 404 3207 "" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:108.0) Gecko/20100101 Firefox/108.0" -
Inside the docker gitlab container this is the proxy log.
Tafka
January 10, 2023, 1:57pm
4
gitlab_master:
image: 'gitlab/gitlab-ce:15.6.2-ce.0'
restart: always
hostname: 'REDACTD'
environment:
GITLAB_OMNIBUS_CONFIG: |
external_url 'https://REDACTED'
nginx['listen_port'] = 80
nginx['listen_https'] = false
nginx['hide_server_tokens'] = 'on'
nginx['gzip_enabled'] = false
nginx['status'] = {
"options" => {
"access_log" => "on"
}
}
nginx['redirect_http_to_https'] = true
registry_nginx['redirect_http_to_https'] = true
gitlab_rails['allowed_hosts'] = ['REDACTED']
gitlab_rails['time_zone'] = 'REDACTED'
gitlab_rails['impersonation_enabled'] = false
gitlab_rails['gravatar_enabled'] = false
gitlab_rails['ldap_enabled'] = true
gitlab_rails['prevent_ldap_sign_in'] = false
gitlab_rails['gitlab_username_changing_enabled'] = false
gitlab_rails['gitlab_default_can_create_group'] = true
gitlab_rails['ldap_servers'] = {
REDACTED
}
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = 'REDACTED'
gitlab_rails['smtp_port'] = 25
gitlab_rails['smtp_domain'] = 'REDACTED'
gitlab_rails['smtp_pool'] = true
gitlab_rails['gitlab_email_enabled'] = true
gitlab_rails['gitlab_email_display_name'] = 'REDACTED'
gitlab_rails['gitlab_email_from'] = 'REDACTED'
gitlab_rails['gitlab_email_reply_to'] = 'REDACTED'
gitlab_rails['incoming_email_enabled'] = false
gitlab_rails['content_security_policy'] = {
'enabled' => true,
'report_only' => false,
'directives' => {
'default_src' => "'self'",
'script_src' => "'self' 'unsafe-inline' 'unsafe-eval'",
'frame_ancestors' => "'self'",
'frame_src' => "'self'",
'img_src' => "* data: blob:",
'style_src' => "'self' 'unsafe-inline'"
}
}
gitlab_rails['rack_attack_git_basic_auth'] = {
'enabled' => true,
'ip_whitelist' => ['127.0.0.1'],
'maxretry' => 10,
'findtime' => 60,
'bantime' => 3600
}
letsencrypt['enable'] = false
gitlab_rails['dependency_proxy_enabled'] = true
gitlab_rails['terraform_state_enabled'] = false
gitlab_rails['packages_enabled'] = true
gitlab_rails['external_diffs_enabled'] = false
gitlab_rails['lfs_enabled'] = true
gitlab_rails['usage_ping_enabled'] = false
the-garbage-collection-on-schedule
registry['enable'] = true
gitlab_rails['registry_path'] = '/mnt/registry-disk'
registry_external_url 'https://REDACTED:5050'
registry_nginx['listen_port'] = 5050
registry_nginx['listen_https'] = false
registry_nginx['hide_server_tokens'] = 'on'
registry_nginx['gzip_enabled'] = false
# Workaround to avoid registry push retry loops with reverse proxies:
registry['env'] = {
"REGISTRY_HTTP_RELATIVEURLS" => true
}
gitlab_rails['gitlab_default_projects_features_issues'] = true
gitlab_rails['gitlab_default_projects_features_merge_requests'] = true
gitlab_rails['gitlab_default_projects_features_wiki'] = true
gitlab_rails['gitlab_default_projects_features_snippets'] = false
gitlab_rails['gitlab_default_projects_features_builds'] = true
gitlab_rails['gitlab_default_projects_features_container_registry'] = true
gitlab_rails['auto_migrate'] = true
# Enable Kubernetes Agent Server over wss
gitlab_kas['enable'] = true
gitlab_kas['gitlab_address'] = 'REDACTED'
ports:
- '80:80'
- '22:22'
- '127.0.0.1:5051:5050'
- '127.0.0.1:8060:8060'
volumes:
- '/srv/gitlab/config:/etc/gitlab'
- '/srv/gitlab/data:/var/opt/gitlab'
- '/srv/gitlab/logs:/var/log/gitlab'
- '/mnt/registry-disk:/mnt/registry-disk'