Tafka
January 6, 2023, 8:42am
#1
Hi
I have a question regarding the /-/cable endpoint.
We are running a selfhosted CE gitlab instance and our monitoring started to notice that users with vscode gitlab extension got a lot of 404 to the /-/cable endpoint.
As I understand this is the ActionCable util from Puma in Ruby.
Somebody has written about it here, but he uses a instance that does not run in docker.
I did a bit of investiation of the issue. It really seems to be a proxying error. The main proxy configuration looks like this:
ProxyAddHeaders On
RequestHeader add X-Forwarded-Ssl on
RequestHeader set X-Forwarded-Proto "https"
ProxyPass unix:///opt/gitlab/gitlab/tmp/sockets/gitlab-workhorse.socket|
http://127.0.0.1/
ProxyPassReverse unix:///opt/gitlab/gitlab/tmp/sockets/gitlab-
workhorse.socket|http://127.0.0.1/
which results in
Started GET "/-/cable" for $REMOTE_IP at 2022-12-22 14:35:51 +…
The docs only specify amount of workers.
Documentation for GitLab Community Edition, GitLab Enterprise Edition, Omnibus GitLab, and GitLab Runner.
I found this website that exposes all the usable ports.
Documentation for GitLab Community Edition, GitLab Enterprise Edition, Omnibus GitLab, and GitLab Runner.
For Puma it is 8080, because ActionCable is puma, do I need to use port 8080 when forwarding front proxy messages to the container?
my conf:
# ports from docker ps command
0.0.0.0:22->22/tcp, :::22->22/tcp, 0.0.0.0:80->80/tcp, :::80->80/tcp, 443/tcp, 127.0.0.1:8060->8060/tcp, 127.0.0.1:5051->5050/tcp
# nginx conf
server {
listen 80;
server_name REDACTED;
return 301 REDACTED$request_uri;
}
server {
listen 443 ssl http2;
server_name REDACTED;
index index.html index.htm;
client_max_body_size 0;
ssl_certificate /etc/ssl/certs/REDACTED.pem;
ssl_certificate_key /etc/ssl/private/REDACTED.pem;
location / {
proxy_pass http://127.0.0.1:80;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto "https";
}
location /-/kubernetes-agent/ {
proxy_set_header Host $host;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_pass http://127.0.0.1:80;
}
location /-/cable {
proxy_set_header Host $host;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_pass http://127.0.0.1:80;
}
}
EDIT:
Also added one line from /var/log/nginx/access.log
REDACTED.IP - - [06/Jan/2023:12:32:44 +0200] "GET /-/cable HTTP/1.1" 404 1640 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36" "-"
Tafka
January 9, 2023, 9:22am
#2
I tried to use this Nginx configuration that was present here, but to no avail:
Tafka
January 9, 2023, 9:31am
#3
Started GET "/-/cable" for 172.17.0.1 at 2023-01-09 09:29:38 +0000
Started GET "/-/cable/"[non-WebSocket] for 172.17.0.1 at 2023-01-09 09:29:38 +0000
Finished "/-/cable/"[non-WebSocket] for 172.17.0.1 at 2023-01-09 09:29:38 +0000
{"content_type":"text/html; charset=utf-8","correlation_id":"01GPAXTBCEE6QW84QMVDH1MV3F","duration_ms":40,"host":"REDACTED","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"127.0.0.1:0","remote_ip":"127.0.0.1","route":"^/-/","status":404,"system":"http","time":"2023-01-09T09:29:38Z","ttfb_ms":39,"uri":"/-/cable","user_agent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:108.0) Gecko/20100101 Firefox/108.0","written_bytes":3207}
172.17.0.1 - - [09/Jan/2023:09:29:38 +0000] "GET /-/cable HTTP/1.0" 404 3207 "" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:108.0) Gecko/20100101 Firefox/108.0" -
Inside the docker gitlab container this is the proxy log.
Tafka
January 10, 2023, 1:57pm
#4
gitlab_master:
image: 'gitlab/gitlab-ce:15.6.2-ce.0'
restart: always
hostname: 'REDACTD'
environment:
GITLAB_OMNIBUS_CONFIG: |
external_url 'https://REDACTED'
nginx['listen_port'] = 80
nginx['listen_https'] = false
nginx['hide_server_tokens'] = 'on'
nginx['gzip_enabled'] = false
nginx['status'] = {
"options" => {
"access_log" => "on"
}
}
nginx['redirect_http_to_https'] = true
registry_nginx['redirect_http_to_https'] = true
gitlab_rails['allowed_hosts'] = ['REDACTED']
gitlab_rails['time_zone'] = 'REDACTED'
gitlab_rails['impersonation_enabled'] = false
gitlab_rails['gravatar_enabled'] = false
gitlab_rails['ldap_enabled'] = true
gitlab_rails['prevent_ldap_sign_in'] = false
gitlab_rails['gitlab_username_changing_enabled'] = false
gitlab_rails['gitlab_default_can_create_group'] = true
gitlab_rails['ldap_servers'] = {
REDACTED
}
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = 'REDACTED'
gitlab_rails['smtp_port'] = 25
gitlab_rails['smtp_domain'] = 'REDACTED'
gitlab_rails['smtp_pool'] = true
gitlab_rails['gitlab_email_enabled'] = true
gitlab_rails['gitlab_email_display_name'] = 'REDACTED'
gitlab_rails['gitlab_email_from'] = 'REDACTED'
gitlab_rails['gitlab_email_reply_to'] = 'REDACTED'
gitlab_rails['incoming_email_enabled'] = false
gitlab_rails['content_security_policy'] = {
'enabled' => true,
'report_only' => false,
'directives' => {
'default_src' => "'self'",
'script_src' => "'self' 'unsafe-inline' 'unsafe-eval'",
'frame_ancestors' => "'self'",
'frame_src' => "'self'",
'img_src' => "* data: blob:",
'style_src' => "'self' 'unsafe-inline'"
}
}
gitlab_rails['rack_attack_git_basic_auth'] = {
'enabled' => true,
'ip_whitelist' => ['127.0.0.1'],
'maxretry' => 10,
'findtime' => 60,
'bantime' => 3600
}
letsencrypt['enable'] = false
gitlab_rails['dependency_proxy_enabled'] = true
gitlab_rails['terraform_state_enabled'] = false
gitlab_rails['packages_enabled'] = true
gitlab_rails['external_diffs_enabled'] = false
gitlab_rails['lfs_enabled'] = true
gitlab_rails['usage_ping_enabled'] = false
the-garbage-collection-on-schedule
registry['enable'] = true
gitlab_rails['registry_path'] = '/mnt/registry-disk'
registry_external_url 'https://REDACTED:5050'
registry_nginx['listen_port'] = 5050
registry_nginx['listen_https'] = false
registry_nginx['hide_server_tokens'] = 'on'
registry_nginx['gzip_enabled'] = false
# Workaround to avoid registry push retry loops with reverse proxies:
registry['env'] = {
"REGISTRY_HTTP_RELATIVEURLS" => true
}
gitlab_rails['gitlab_default_projects_features_issues'] = true
gitlab_rails['gitlab_default_projects_features_merge_requests'] = true
gitlab_rails['gitlab_default_projects_features_wiki'] = true
gitlab_rails['gitlab_default_projects_features_snippets'] = false
gitlab_rails['gitlab_default_projects_features_builds'] = true
gitlab_rails['gitlab_default_projects_features_container_registry'] = true
gitlab_rails['auto_migrate'] = true
# Enable Kubernetes Agent Server over wss
gitlab_kas['enable'] = true
gitlab_kas['gitlab_address'] = 'REDACTED'
ports:
- '80:80'
- '22:22'
- '127.0.0.1:5051:5050'
- '127.0.0.1:8060:8060'
volumes:
- '/srv/gitlab/config:/etc/gitlab'
- '/srv/gitlab/data:/var/opt/gitlab'
- '/srv/gitlab/logs:/var/log/gitlab'
- '/mnt/registry-disk:/mnt/registry-disk'