Gitlab 14.10.0 - Debian 11 - 404 The page could not be found or you don't have permission to view it

General
, ,
1

Hello!
I’ve recently updated my setup from Debian 10 (Buster) to Debian 11 (Bullseye) and have Gitlab installed.
I’m using Apache instead of Nginx as the system is being used as a website development server which is making use of port 80 and 443.

On my Debian 10 setup I was able to access Gitlab through a proxy / reverse proxy on a subdomain e.g. “git.example.com”
settings can be found below.

However in Debian 11 with the same configuration and setup, I am faced with an error 404 response.

Issue | Error Message
“404 : The page could not be found or you don’t have permission to view it.”

image
image777×673 19.5 KB

Information
Notes:

  • System is running in Oracle Virtual Machine: Virtual Box
  • Example.com is used in place of my DNS / domain
  • Setup uses Subdomain access
  • GitLab has been configured with the use of Apache2 instead of Nginx
  • Docker is NOT being used.
  • Port Forwarding has been configured (Router, VM and UFW Firewall)
    image
    image506×596 45.8 KB

Apache2 Modules

  • headers
  • rewrite
  • ssl
  • proxy
  • proxy_http

Installation
Gitlab CE Packages

curl -s https://packages.gitlab.com/install/repositories/gitlab/gitlab-ce/script.deb.sh | sudo bash

sudo apt-get install gitlab-ce

Gitlab.rb

nano /etc/gitlab/gitlab.rb

## Set Url
external_url "https://gitlab.example.com"

## Give apache user privileges to listen to gitLab
web_server['external_users'] = ['www-data']
web_server['username'] = 'apache' # updated: apache, default: gitlab-www
web_server['group'] = 'apache' # updated: apache, default: gitlab-www

## Disable nginx
nginx['enable'] = false

## Config network
gitlab_workhorse['enable'] = true
gitlab_workhorse['listen_network'] = "tcp"
gitlab_workhorse['listen_umask'] = 0
gitlab_workhorse['listen_addr'] = "127.0.0.1:8181"
gitlab_workhorse['auth_backend'] = "http://127.0.0.1:8080" 

# Puma
puma['listen'] = '127.0.0.1'
puma['port'] = 8080

Apache Http.conf

nano /etc/apache2/sites-available/http.conf

<VirtualHost *:80>
  ServerName git.example.com
  ServerSignature Off

  RewriteEngine on
  RewriteCond %{HTTPS} !=on
  RewriteRule .* https://%{SERVER_NAME}%{REQUEST_URI} [NE,R,L]
</VirtualHost>

Apache Https.conf

nano /etc/apache2/sites-available/https.conf
certbot --apache

<IfModule mod_ssl.c>
<VirtualHost *:443>

    # Server
    ServerName git.example.com
    ServerSignature Off

    ProxyPreserveHost On

    # Ensure that encoded slashes are not decoded but left in their encoded state.
    # http://doc.gitlab.com/ce/api/projects.html#get-single-project
    AllowEncodedSlashes NoDecode

    <Location />
        # New authorization commands for apache 2.4 and up
        # http://httpd.apache.org/docs/2.4/upgrading.html#access
        Require all granted

        # Allow forwarding to gitlab-workhorse
        ProxyPassReverse http://127.0.0.1:8181
        ProxyPassReverse http://git.example.com
    </Location>

    # Apache equivalent of nginx try files
    # http://serverfault.com/questions/290784/what-is-apaches-equivalent-of-nginxs-try-files
    # http://stackoverflow.com/questions/10954516/apache2-proxypass-for-rails-app-gitlab
    RewriteEngine on

    # Forward all requests to gitlab-workhorse except existing files like error documents
    RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_FILENAME} !-f [OR]
    RewriteCond %{REQUEST_URI} ^/uploads/.*
    RewriteRule .* http://127.0.0.1:8181%{REQUEST_URI} [P,QSA,NE] 
    

    RequestHeader set X_FORWARDED_PROTO 'https'
    RequestHeader set X-Forwarded-Ssl on

    # Needed for downloading attachments
    DocumentRoot /opt/gitlab/embedded/service/gitlab-rails/public

    #Set up apache error documents, if back end goes down (i.e. 503 error) then a maintenance/deploy page is thrown up.
    ErrorDocument 404 /404.html
    ErrorDocument 422 /422.html
    ErrorDocument 500 /500.html
    ErrorDocument 502 /502.html
    ErrorDocument 503 /503.html

    # The default directory log is usually /var/log/httpd
    # As this is an Apache2 configuration, use of ${APACHE_LOG_DIR} may be more approrpaite
    LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b" common_forwarded
    ErrorLog ${APACHE_LOG_DIR}/git.example.com_error.log
    CustomLog ${APACHE_LOG_DIR}/git.example.com_forwarded.log common_forwarded
    CustomLog ${APACHE_LOG_DIR}/git.example.com_access.log combined env=!dontlog
    CustomLog ${APACHE_LOG_DIR}/git.example.com.log combined

    # SSL
    # Use strong encryption ciphers only
    # See ciphers(1) http://www.openssl.org/docs/apps/ciphers.html
    SSLEngine on
    SSLProtocol all -SSLv2
    SSLHonorCipherOrder on
    SSLCipherSuite "ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS"
    Header add Strict-Transport-Security: "max-age=15768000;includeSubdomains"
    SSLCompression Off

    SSLCertificateFile /etc/letsencrypt/live/git.example.com/fullchain.pem
    SSLCertificateKeyFile /etc/letsencrypt/live/git.example.com/privkey.pem
    Include /etc/letsencrypt/options-ssl-apache.conf
</VirtualHost>
</IfModule>

System & Gitlab

gitlab-rake gitlab:env:info

System:         Debian 11
Current User:   git
Using RVM:      no
Ruby Version:   2.7.5p203
Gem Version:    3.1.4
Bundler Version:2.2.33
Rake Version:   13.0.6
Redis Version:  6.2.6
Sidekiq Version:6.4.0
Go Version:     unknown

GitLab information
Version:        14.10.0
Revision:       88da5554d96
Directory:      /opt/gitlab/embedded/service/gitlab-rails
DB Adapter:     PostgreSQL
DB Version:     12.7
URL:            https://git.example.com
HTTP Clone URL: https://git.example.com/some-group/some-project.git
SSH Clone URL:  git@git.example.com:some-group/some-project.git
Using LDAP:     no
Using Omniauth: yes
Omniauth Providers:
GitLab Shell
Version: 13.25.1
Repository storage paths:
- default: /var/opt/gitlab/git-data/repositories
GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell

Gitlab Status

service gitlab-runsvdir status

gitlab-runsvdir.service - GitLab Runit supervision process
     Loaded: loaded (/lib/systemd/system/gitlab-runsvdir.service; enabled; vendor preset: enabled)
     Active: active (running) since Sun 2022-05-01 16:48:52 BST; 40min ago
   Main PID: 1786 (runsvdir)
      Tasks: 304 (limit: 4915)
     Memory: 2.8G
        CPU: 43min 49.508s
     CGroup: /system.slice/gitlab-runsvdir.service
             ├─1786 runsvdir -P /opt/gitlab/service log: ..............................................................>
             ├─1788 runsv node-exporter
             ├─1789 runsv grafana
             ├─1790 runsv gitlab-workhorse
             ├─1791 runsv sidekiq
             ├─1792 runsv gitlab-kas
             ├─1793 runsv puma
             ├─1794 runsv prometheus
             ├─1795 runsv redis
             ├─1796 runsv postgresql
             ├─1797 runsv postgres-exporter
             ├─1798 runsv logrotate
             ├─1799 runsv redis-exporter
             ├─1800 runsv alertmanager
             ├─1801 runsv gitlab-exporter
             ├─1802 runsv gitaly
             ├─1803 svlogd -tt /var/log/gitlab/node-exporter
             ├─1804 svlogd -tt /var/log/gitlab/prometheus
             ├─1805 svlogd /var/log/gitlab/gitlab-workhorse
             ├─1806 svlogd -tt /var/log/gitlab/gitlab-kas
             ├─1807 svlogd -tt /var/log/gitlab/puma
             ├─1808 svlogd -tt /var/log/gitlab/redis
             ├─1809 svlogd /var/log/gitlab/sidekiq
             ├─1810 svlogd -tt /var/log/gitlab/logrotate
             ├─1811 /opt/gitlab/embedded/bin/node_exporter --web.listen-address=localhost:9100 --collector.mountstats ->
             ├─1812 svlogd -tt /var/log/gitlab/grafana
             ├─1813 svlogd -tt /var/log/gitlab/gitlab-exporter
             ├─1814 svlogd -tt /var/log/gitlab/postgresql
             ├─1815 svlogd -tt /var/log/gitlab/redis-exporter
             ├─1816 svlogd -tt /var/log/gitlab/alertmanager
             ├─1817 svlogd -tt /var/log/gitlab/postgres-exporter
             ├─1818 svlogd /var/log/gitlab/gitaly
             ├─1819 /opt/gitlab/embedded/bin/prometheus --web.listen-address=localhost:9090 --storage.tsdb.path=/var/op>
             ├─1820 /opt/gitlab/embedded/bin/gitlab-workhorse -listenNetwork tcp -listenUmask 0 -listenAddr localhost:8>
             ├─1821 /opt/gitlab/embedded/bin/gitlab-kas --configuration-file /var/opt/gitlab/gitlab-kas/gitlab-kas-conf>
             ├─1823 /bin/sh /opt/gitlab/embedded/bin/gitlab-logrotate-wrapper
             ├─1824 /opt/gitlab/embedded/bin/redis-server unixsocket:/var/opt/gitlab/redis/redis.socket
             ├─1825 ruby /opt/gitlab/embedded/service/gitlab-rails/bin/sidekiq-cluster -e production -r /opt/gitlab/emb>
             ├─1833 /opt/gitlab/embedded/bin/postgres -D /var/opt/gitlab/postgresql/data
             ├─1834 /opt/gitlab/embedded/bin/redis_exporter --web.listen-address=localhost:9121 --redis.addr=unix:///va>
             ├─1835 /opt/gitlab/embedded/bin/alertmanager --web.listen-address=localhost:9093 --storage.path=/var/opt/g>
lines 1-49

Debugging
As this is system running in a VM, I’ve made restore points to recover the VM and rollback if necessary.

Attempts made

  • Change folder permissions and ownership of gitlab.rb & /opt/gitlab/embedded/service/gitlab-rails/public
  • Checked examples of Apache Configurations from various sources
  • Cleared Cache / Cookies in browser
  • Used other browsers to attempt loading
  • Tried loading Gitlab through localhost on Debian server directly “same result as accessing via git.example.com”