Timeout on fresh install of gitlab-CE

KurtSys-ftrprf · August 3, 2023, 10:30am

I have a fresh install of Gitlab CE on a (fresh) Ubuntu server on azure. I followed these steps:

When running gitlab-ctl reconfigure, there are no errors and when checking the status, all seems fine:

gitlab-ctl status
run: alertmanager: (pid 25974) 1118s; run: log: (pid 16605) 8207s
run: crond: (pid 25982) 1117s; run: log: (pid 19799) 6955s
run: gitaly: (pid 25990) 1117s; run: log: (pid 16029) 8325s
run: gitlab-exporter: (pid 26003) 1117s; run: log: (pid 16495) 8225s
run: gitlab-kas: (pid 26159) 1104s; run: log: (pid 16241) 8306s
run: gitlab-workhorse: (pid 26090) 1107s; run: log: (pid 16402) 8242s
run: logrotate: (pid 26100) 1106s; run: log: (pid 15981) 8335s
run: nginx: (pid 26106) 1106s; run: log: (pid 16426) 8237s
run: node-exporter: (pid 26120) 1106s; run: log: (pid 16476) 8231s
run: postgres-exporter: (pid 26125) 1105s; run: log: (pid 16671) 8199s
run: postgresql: (pid 26139) 1105s; run: log: (pid 16101) 8314s
run: puma: (pid 26148) 1104s; run: log: (pid 16325) 8254s
run: redis: (pid 26153) 1104s; run: log: (pid 16007) 8329s
run: redis-exporter: (pid 26168) 1103s; run: log: (pid 16522) 8217s
run: registry: (pid 26175) 1104s; run: log: (pid 19818) 6950s
run: sidekiq: (pid 26184) 1104s; run: log: (pid 16362) 8249s

It seems gitlab is running fine: I can access the login page when I run wget localhost:8080 on the VM (see below).

However, when I try to access gitlab by url (or ip address), the requests times out.

# wget localhost
--2023-08-03 10:23:19--  http://localhost/
Resolving localhost (localhost)... 127.0.0.1
Connecting to localhost (localhost)|127.0.0.1|:80... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: https://cf-git.westeurope.cloudapp.azure.com:443/ [following]
--2023-08-03 10:23:19--  https://cf-git.westeurope.cloudapp.azure.com/
Resolving cf-git.westeurope.cloudapp.azure.com (cf-git.westeurope.cloudapp.azure.com)... 4.210.152.225
Connecting to cf-git.westeurope.cloudapp.azure.com (cf-git.westeurope.cloudapp.azure.com)|4.210.152.225|:443... ^C

The resolving localhost and ip address works fine: it connects to https://cf-git.westeurope.cloudapp.azure.com/ . However, there’s no answer when accessing that url.

My best guess is that nginx is not properly configured. I have no custom/own nginx installed, so I think it’s the nginx config of gitlab. Not sure if it’s the right one, but checking the nginx configuration of gitlab (removing all the commented lines), nginx only seems to listen to port 80:

/opt/gitlab/embedded/conf# cat nginx.conf

worker_processes  1;

events {
    worker_connections  1024;
}

http {
    include       mime.types;
    default_type  application/octet-stream;

    sendfile        on;

    keepalive_timeout  65;

    server {
        listen       80;
        server_name  localhost;

        location / {
            root   html;
            index  index.html index.htm;
        }

        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   html;
        }
    }
}

What am I missing? Or rather, how can I debug/solve the problem?

Thanks a lot!

$ wget localhost:8080
--2023-08-03 10:20:13--  http://localhost:8080/
Resolving localhost (localhost)... 127.0.0.1
Connecting to localhost (localhost)|127.0.0.1|:8080... connected.
HTTP request sent, awaiting response... 302 Found
Location: http://localhost:8080/users/sign_in [following]
--2023-08-03 10:20:13--  http://localhost:8080/users/sign_in
Connecting to localhost (localhost)|127.0.0.1|:8080... connected.
HTTP request sent, awaiting response... 200 OK
Length: 8175 (8.0K) [text/html]
Saving to: ‘index.html’

index.html                        100%[==========================================================>]   7.98K  --.-KB/s    in 0s      

2023-08-03 10:20:13 (1.06 GB/s) - ‘index.html’ saved [8175/8175]

root@vm-cod-p-git:/opt/gitlab/embedded/conf# cat index.html 
<!DOCTYPE html>
<html class="html-devise-layout" lang="en">
<head prefix="og: http://ogp.me/ns#">
<meta charset="utf-8">

...

<form class="gl-p-5 gl-show-field-errors js-arkose-labs-form" id="new_user" aria-live="assertive" data-testid="sign-in-form" action="/users/sign_in" accept-charset="UTF-8" method="post"><input type="hidden" name="authenticity_token" value="_cvkxI6c7hlFQtobdca39Ml1tXxgnnBMbfvGQf_Mpm8IbpAn49mqqSqOQIpgtxrre_87-g9_2MR9bF1WF4BzsA" autocomplete="off" /><div class="form-group">
<label for="user_login">Username or email</label>
<input class="form-control gl-form-input js-username-field" autocomplete="username" autofocus="autofocus" autocapitalize="off" autocorrect="off" required="required" title="This field is required." data-qa-selector="login_field" data-testid="username-field" type="text" name="user[login]" id="user_login" />
</div>
<div class="form-group">
<label for="user_password">Password</label>
<input class="form-control gl-form-input js-password" data-id="user_password" data-qa-selector="password_field" data-testid="password-field" data-name="user[password]" type="password" name="user[password]" id="user_password" />
<div class="form-text gl-text-right">
<a href="/users/password/new">Forgot your password?</a>
</div>
</div>
<div class="form-group">
</div>
<div class="gl-form-checkbox custom-control custom-checkbox">
<input name="user[remember_me]" type="hidden" value="0" autocomplete="off" /><input autocomplete="off" class="custom-control-input" type="checkbox" value="1" name="user[remember_me]" id="user_remember_me" />
<label class="custom-control-label" for="user_remember_me"><span>Remember me</span></label>
</div>

<button class="gl-button btn btn-block btn-md btn-confirm js-sign-in-button " data-qa-selector="sign_in_button" data-testid="sign-in-button" type="submit"><span class="gl-button-text">
Sign in

</span>

</button></form>

KurtSys-ftrprf · August 3, 2023, 10:42am

the actual config nginx config files of gitlab:

/var/opt/gitlab/nginx/conf# ls
gitlab-health.conf  gitlab-http.conf  gitlab-registry.conf  nginx.conf  nginx-status.conf

/var/opt/gitlab/nginx/conf# cat gitlab-http.conf 
# This file is managed by gitlab-ctl. Manual changes will be
# erased! To change the contents below, edit /etc/gitlab/gitlab.rb
# and run `sudo gitlab-ctl reconfigure`.

## GitLab
## Modified from https://gitlab.com/gitlab-org/gitlab-foss/blob/master/lib/support/nginx/gitlab-ssl & https://gitlab.com/gitlab-org/gitlab-foss/blob/master/lib/support/nginx/gitlab
##
## Lines starting with two hashes (##) are comments with information.
## Lines starting with one hash (#) are configuration parameters that can be uncommented.
##
##################################
##        CHUNKED TRANSFER      ##
##################################
##
## It is a known issue that Git-over-HTTP requires chunked transfer encoding [0]
## which is not supported by Nginx < 1.3.9 [1]. As a result, pushing a large object
## with Git (i.e. a single large file) can lead to a 411 error. In theory you can get
## around this by tweaking this configuration file and either:
## - installing an old version of Nginx with the chunkin module [2] compiled in, or
## - using a newer version of Nginx.
##
## At the time of writing we do not know if either of these theoretical solutions works.
## As a workaround users can use Git over SSH to push large files.
##
## [0] https://git.kernel.org/cgit/git/git.git/tree/Documentation/technical/http-protocol.txt#n99
## [1] https://github.com/agentzh/chunkin-nginx-module#status
## [2] https://github.com/agentzh/chunkin-nginx-module
##
###################################
##         configuration         ##
###################################

## Redirects all HTTP traffic to the HTTPS host
server { ## HTTPS redirect server
  listen *:80;

  server_name cf-git.westeurope.cloudapp.azure.com;
  server_tokens off; ## Don't show the nginx version number, a security best practice

  location /.well-known/acme-challenge/ {
    root /var/opt/gitlab/nginx/www/;
  }

  location / {
    return 301 https://cf-git.westeurope.cloudapp.azure.com:443$request_uri;
  }

  # Pass headers because we are serving monitoring endpoints directly without
  # redirection
  proxy_set_header Host $http_host_with_default;
  proxy_set_header X-Real-IP $remote_addr;
  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  proxy_set_header Upgrade $http_upgrade;
  proxy_set_header Connection $connection_upgrade;
  proxy_set_header X-Forwarded-Ssl on;
  proxy_set_header X-Forwarded-Proto https;

  # health checks configuration
  include /var/opt/gitlab/nginx/conf/gitlab-health.conf;

  access_log  /var/log/gitlab/nginx/gitlab_access.log gitlab_access;
  error_log   /var/log/gitlab/nginx/gitlab_error.log error;
} ## end HTTPS redirect server

server { ## HTTPS server
  listen *:443 ssl http2;


  server_name cf-git.westeurope.cloudapp.azure.com;
  server_tokens off; ## Don't show the nginx version number, a security best practice

  ## Increase this if you want to upload large attachments
  ## Or if you want to accept large git objects over http
  client_max_body_size 0;

  ## Strong SSL Security
  ## https://raymii.org/s/tutorials/Strong_SSL_Security_On_nginx.html & https://cipherli.st/
  ssl_certificate /etc/gitlab/ssl/vm-cod-p-git.hnfu5efvkcsuhgzqkzzamcxiib.ax.internal.cloudapp.net.crt;
  ssl_certificate_key /etc/gitlab/ssl/vm-cod-p-git.hnfu5efvkcsuhgzqkzzamcxiib.ax.internal.cloudapp.net.key;

  # GitLab needs backwards compatible ciphers to retain compatibility with Java IDEs
  ssl_ciphers 'ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384';
  ssl_protocols  TLSv1.2 TLSv1.3;
  ssl_prefer_server_ciphers off;
  ssl_session_cache  shared:SSL:10m;
  ssl_session_tickets off;
  ssl_session_timeout  1d;



  ## Real IP Module Config
  ## http://nginx.org/en/docs/http/ngx_http_realip_module.html

  ## HSTS Config
  ## https://www.nginx.com/blog/http-strict-transport-security-hsts-and-nginx/
  add_header Strict-Transport-Security "max-age=63072000";

  # Rails sets a default policy of strict-origin-when-cross-origin, so
  # hide that and just send the one we've configured for nginx
  proxy_hide_header Referrer-Policy;
  add_header Referrer-Policy strict-origin-when-cross-origin;

  ## Individual nginx logs for this GitLab vhost
  access_log  /var/log/gitlab/nginx/gitlab_access.log gitlab_access;
  error_log   /var/log/gitlab/nginx/gitlab_error.log error;

  if ($http_host = "") {
    set $http_host_with_default "cf-git.westeurope.cloudapp.azure.com";
  }

  if ($http_host != "") {
    set $http_host_with_default $http_host;
  }

  gzip on;
  gzip_static on;
  gzip_comp_level 2;
  gzip_http_version 1.1;
  gzip_vary on;
  gzip_disable "msie6";
  gzip_min_length 250;
  gzip_proxied no-cache no-store private expired auth;
  gzip_types text/plain text/css application/x-javascript text/xml application/xml application/xml+rss text/javascript application/json;

  ## https://github.com/gitlabhq/gitlabhq/issues/694
  ## Some requests take more than 30 seconds.
  proxy_read_timeout      3600;
  proxy_connect_timeout   300;
  proxy_redirect          off;
  proxy_http_version 1.1;

  proxy_set_header Host $http_host_with_default;
  proxy_set_header X-Real-IP $remote_addr;
  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  proxy_set_header Upgrade $http_upgrade;
  proxy_set_header Connection $connection_upgrade;
  proxy_set_header X-Forwarded-Ssl on;
  proxy_set_header X-Forwarded-Proto https;

  location ~ (/api/v\d/jobs/\d+/artifacts$|/import/gitlab_project$|\.git/git-receive-pack$|\.git/gitlab-lfs/objects|\.git/info/lfs/objects/batch$) {
    proxy_cache off;
    proxy_pass http://gitlab-workhorse;
    proxy_request_buffering off;
  }

  location ~ ^/api/v\d {
    proxy_cache off;
    proxy_pass http://gitlab-workhorse;
    proxy_intercept_errors off;
  }


  location = /-/kubernetes-agent/ {
    proxy_pass http://localhost:8150/;
    proxy_intercept_errors off;
  }

  location /-/kubernetes-agent/k8s-proxy/ {
    proxy_pass http://localhost:8154/;
    proxy_buffering off;
    proxy_intercept_errors off;
  }

  # health checks configuration
  include /var/opt/gitlab/nginx/conf/gitlab-health.conf;

  location /.well-known/acme-challenge/ {
    root /var/opt/gitlab/nginx/www/;
  }

  location / {
    proxy_cache off;
    proxy_pass  http://gitlab-workhorse;
  }

  location /assets {
    add_header X-Content-Type-Options nosniff;
    proxy_cache gitlab;
    proxy_pass  http://gitlab-workhorse;
  }

  error_page 404 /404.html;
  error_page 500 /500.html;
  error_page 502 /502.html;
  location ~ ^/(404|500|502)(-custom)?\.html$ {
    root /opt/gitlab/embedded/service/gitlab-rails/public;
    internal;
  }

  
} ## end HTTPS server

iwalker · August 3, 2023, 12:38pm

Have you checked to ensure that Azure ingress rules have been created to allow traffic for ports 443 and 80? Usually, Amazon, Azure, IBM, etc have traffic blocked until you create ingress rules to allow the access.

Also, all config for nginx is controlled in /etc/gitlab/gitlab.rb - so you shouldn’t be editing or changing the files in /opt or /var/opt/gitlab manually. All changes are always applied to /etc/gitlab/gitlab.rb and then activating by running gitlab-ctl reconfigure

KurtSys-ftrprf · August 3, 2023, 12:55pm

Absolutely right: I needed to add inbound rules for http and https!

(I didn’t change anything to the config manually, I was just trying to figure out what was wrong )

Thanks a lot!