Gitlab Behind a Apache Revers proxy with SSL

Hey

I have two servers on the same network, server “A” is the Web server for our lan party website that is running Debian 9.4 with Apache2 and gets SSL certificate from LetsEncrypt.
Server “A” is also responsible to proxy every request to server “B” and upgrade the connection to use HTTPS

Server “B” is running Ubuntu 18.04 a dedicated Gitlab server (In a Docker container) for development for the website and other tools we use on the party

Every thing works fine when i use HTTPS but,
The problem begins when i try to setup HTTPS then i always gets

> Service Unavailable
> The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later.

What i have tried til now is

I Have read this site to try to enable HTTPS

• https://docs.gitlab.com/omnibus/settings/nginx.html#supporting-proxied-ssl

Using this config file on Apache2

• https://gitlab.com/gitlab-org/gitlab-recipes/blob/master/web-server/apache/gitlab-omnibus-ssl-apache24.conf

tried to change diffrent headers on the proxy

• https://gitlab.com/gitlab-org/gitlab-recipes/blob/master/web-server/apache/gitlab-omnibus-ssl-apache24.conf

And much more

I had a similar issue which I resolved but I’m currently out on the road so can’t post my configs.

Will try to do so later today.

FYI my install is a gitlab-ce on Devuan.

Here is the apache2 sites-enabled config. Note that I use 8443 as the external https port.

<VirtualHost *:8443>

    ServerName gitlab.mydomain.com
    SSLEngine on
    SSLCACertificateFile /etc/gitlab/trusted-certs/cert.pem
    SSLCertificateFile /etc/gitlab/trusted-certs/fullchain.pem
    SSLCertificateKeyFile /etc/gitlab/trusted-certs/privkey.pem
<Proxy *>
    Require all granted
</Proxy>

SSLProxyEngine on
ProxyRequests Off
ProxyPass / https://gitlab.mydomain.com:4443/
ProxyPassReverse / https://gitlab.mydomain.com/

Header edit Location ^http://gitlab.mydomain.com/ https://gitlab.domain.com/
RequestHeader set X-Forwarded-Proto "https"

Here are the relevant bits in gitlab.rb

external_url ‘https://gitlab.mydomain.com:4443’
nginx[‘ssl_client_certificate’] = “/etc/gitlab/trusted-certs/cert.pem”
nginx[‘ssl_certificate’]= “/etc/gitlab/trusted-certs/fullchain.pem”
nginx[‘ssl_certificate_key’] = “/etc/gitlab/trusted-certs/privkey.pem”

Here are the scp commands from the Letsencrypt server to the gitlab box.

You will need a gitlab-ctl reconfigure so that gitlab picks the certs up and generates the corect links.

Note I also use a non standard ssh port.

scp -P 2211 /etc/dehydrated/certs/mydomain.com/fullchain.pem root@gitlab.mydomain.com://etc/gitlab/trusted-certs/fullchain.pem
scp -P 2211 /etc/dehydrated/certs/mydomain.com/privkey.pem root@gitlab.mydomain.com://etc/gitlab/trusted-certs/privkey.pem
scp -P 2211 /etc/dehydrated/certs/mydomain.com/cert.pem root@gitlab.mydomain.com://etc/gitlab/trusted-certs/cert.pem

Hope that gets you running - took me days to figure this lot out.

Essentially you leave the nginx webserver running and proxy external->apache->nginx->gitlab

You CAN dispense with nginx, but then you need to proxy direct to the various required ports - see the gitlab nginx confs if you want to see how. I decided it was easier to leave the CE nginx install in place and running.

Note also that I don’t run any http only access to the gitlab box.

Hope that helps…

Super duper much thank you <3
I just needet to add

RewriteEngine On
# This will enable the Rewrite capabilities

RewriteCond %{HTTPS} !=on
# This checks to make sure the connection is not already HTTPS

RewriteRule ^/?(.*) https://%{SERVER_NAME}/$1 [R,L]

https://wiki.apache.org/httpd/RewriteHTTPToHTTPS

Thank you again, I have now struggled wtih this problem over a month >.<

Hi @xaner4 and @reetp

I am struggling with this issue and have been for weeks. I reached the web set up and went through all of the web set up but could never reach the gitlab CE using the my subdomain. I finally decided to uninstall gitlab and start from scratch. I posted in this forum but have not received any comment.

How is your system working? And what advise, from your experience, do you have as I proceed with this solution?

Is it possible to post other relevant pieces of gitlab.rb, please?

Is it true the <VirtualHost *:4443> resides on the reverse proxy server which is also called apache2 sites-enabled config and Letsencrypt server?

What is the or the apache2 configuration on the gitlab box? What is meant by “Note also that I don’t run any http only access to the gitlab box.”

Thanks!

I dont remember much of how I did it right now but her is the only part of the gitlab.rb files that is related to this.

    external_url 'https://gitlab.domain.com'
    gitlab_rails['trusted_proxies'] = ["172.16.1.50"] # The local aadress from where apache is
    nginx['ssl_client_certificate'] = "/etc/gitlab/trusted-certs/cert.pem"
    nginx['ssl_certificate'] = "/etc/gitlab/trusted-certs/fullchain.pem"
    nginx['ssl_certificate_key'] = "/etc/gitlab/trusted-certs/privkey.pem"

You need to have the SSL certificate on the Gitlab server too.

Thanks @xaner4

I am reaching the gitlab server which is of course behind the apache reverse proxy but the directory is being shown.
For the gitlab server the DocumentRoot is:

/opt/gitlab/embedded/services/gitlab-rails/public

What is the correct DocumentRoot for the gitlab server?

Is that DocumentRoot pointing to a place on the Apache server? or is apache installed on the gitlab server?

x - client, computer on internet, user
y - reverse proxy
z - gitlab computer

x —> y —> z —>

or

Client --> Reverse Proxy Server      ---------------------->  Gitlab Server
           
x  --- >    y = < VirtualHost  *: 443 >  -------------------> z =   <VirtualHost * : 80>
                   ServerName y                                        ServerName x
                   ProxyPass / http:// z                               DocumentRoot ???????
                   ReverseProxyPass / http:// z                        ( other directives )                   
                 < / VirturalHost>                                  < / VirturalHost>

z = GitLab Server DocumentRoot: /opt/gitlab/embedded/service/gitlab-rails/public (???)

Apache runs on y = Reverse Proxy Server. The Reverse Proxy Server ProxyPass/ReverseProxyPass to z = GitLab Server. The GitLab Server serves up GitLab for x = client

The DocumentRoot is pointing to a place on the GitLab Server running Apache.

Hmmm, I don’t know the DocumentRoot to use on the “Z” host.

My setup is a little different,

A - Client
B - Apache revers Proxy
C - Nginx proxy (Provided by Gitlab-omnibus install on docker)
D - Gitlab

Thanks @xaner4. The set up for the GitLab Server was found here:

The only modification I made was to copy /opt to where web services are served from and chown -R www-data:www-data opt

After I lost my own config file this weekend after a server failure (Without a backup of the apache config files, Luckily I had a backup of everything else that was important) and many hours of frustration of what I did to make it work the first time around.
so I will publish my apache config file for later reference for myself and a sort of backup of it so I can find it again XD (I will ofc do a backup of it to another place also)

    <VirtualHost *:80>
        ServerName git.example.com
        ServerSignature Off
        RewriteEngine on
        RewriteCond %{HTTPS} !=on
        RewriteRule .* https://%{SERVER_NAME}%{REQUEST_URI} [NE,R,L]
</VirtualHost>

<VirtualHost *:443>
    # Added by Letsencrypt
    SSLEngine on
    SSLCertificateFile /etc/letsencrypt/live/git.example.com/fullchain.pem
    SSLCertificateKeyFile /etc/letsencrypt/live/git.example.com/privkey.pem
    Include /etc/letsencrypt/options-ssl-apache.conf


    ServerName git.example.com
    ProxyPreserveHost On
                                                                                                                                                                                      
    RewriteEngine On
    # This will enable the Rewrite capabilities                                                                                                                                                                                                                                       
    RewriteCond %{HTTPS} !=on
    # This checks to make sure the connection is not already HTTPS                                                                                                                                                                                                                    
    RewriteRule ^/?(.*) https://%{SERVER_NAME}/$1 [R,L]
                                                                                                                                                                                                
    RequestHeader set X-Forwarded-Proto "https"
    RequestHeader set X-Forwarded-Ssl on

    SSLProxyEngine on
    ProxyRequests on
    ProxyPass / https://172.16.1.20/ nocanon
    ProxyPassReverse / https://172.16.1.20/

    # Ensure that encoded slashes are not decoded but left in their encoded state.                                                                                                                                                                                                    
    # http://doc.gitlab.com/ce/api/projects.html#get-single-project                                                                                                                                                                                                                   
    AllowEncodedSlashes NoDecode

    # Show Apache where Error Documents is                                                                                                                                                                                                                                            
    DocumentRoot /var/www/gitlab/
    #Set up apache error documents, if back end goes down (i.e. 503 error) then a maintenance/deploy page is thrown up.                                                                                                                                                               
    ErrorDocument 404 /404.html
    ErrorDocument 422 /422.html
    ErrorDocument 500 /500.html
    ErrorDocument 502 /502.html
    ErrorDocument 503 /503.html


    # Log folder /var/log/apache2.                                                                                                                                                                                                                                                    
    LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b" common_forwarded
    ErrorLog ${APACHE_LOG_DIR}/git.example.com_error.log
    CustomLog ${APACHE_LOG_DIR}/git.example.com_forwarded.log common_forwarded
    CustomLog ${APACHE_LOG_DIR}/git.example.com_access.log combined env=!dontlog
    CustomLog ${APACHE_LOG_DIR}/git.example.com.log combined

</VirtualHost>

# Is used for Docker container registry
<VirtualHost *:5005>
    SSLEngine on
    SSLCertificateFile /etc/letsencrypt/live/git.example.com/fullchain.pem
    SSLCertificateKeyFile /etc/letsencrypt/live/git.example.com/privkey.pem
    Include /etc/letsencrypt/options-ssl-apache.conf

    ServerName git.example.com
    ProxyPreserveHost On

    RequestHeader set X-Forwarded-Proto "https"
    RequestHeader set X-Forwarded-SSL on

    SSLProxyEngine on
    ProxyRequests on
    ProxyPass / https://172.16.1.20:5005/ nocanon
    ProxyPassReverse / https://172.16.1.20:5005/
</VirtualHost>

If some one have more experience with apache config and have any improvment to speed and security around it please let me know what to do better

gitlab.rb config

external_url 'https://git.example.com'
gitlab_rails['trusted_proxies'] = ["172.16.1.50"]

nginx['ssl_client_certificate'] = "/etc/gitlab/trusted-certs/cert.pem"
nginx['ssl_certificate'] = "/etc/gitlab/trusted-certs/fullchain.pem"
nginx['ssl_certificate_key'] = "/etc/gitlab/trusted-certs/privkey.pem"

gitlab_rails['registry_enabled'] = true
gitlab_rails['registry_port'] = "5005"
registry['enable'] = true
registry_nginx['enable'] = true
registry_nginx['listen_port'] = 5005
registry_nginx['listen_https'] = true
registry_nginx['ssl_certificate'] = "/etc/gitlab/trusted-certs/fullchain.pem"
registry_nginx['ssl_certificate_key'] = "/etc/gitlab/trusted-certs/privkey.pem"

Hi @reetp! I deployed a GitLab via docker. The host machine has Apache installed so I did configured to reverse proxy like this

<VirtualHost *:443>
  ServerName repo.company.com
  SetEnv HTTPS on
  <Location />
    Order allow,deny
    Allow from all
    Require all granted
  </Location>
  ProxyPass / http://localhost:5221/
  ProxyPassReverse / http://localhost:55221/
</VirtualHost>

GitLab uses the following port 5221:80 5223:443 and the external url matches with the apache config repo.company.com. This works fine but I would like to enable SSL inside the GitLab docker container. I did enabled the letsencrypt but I’m not sure how to reverseproxy with its https.

@iryanjan18 Thanks for jumping in with your expertise! I am eager to hear if this help @reetp get any closer to a solution!

The answer is it is damn tricky and in the end I didn’t have the time to waste on making it work correctly.

Most of it worked OK, but the web IDE/editor never did - it needs some more trickery to work.

In the end I just ran gitlab on a different port and linked it to use my existing letsencrypt certificates.

Like a lot of stuff in gitlab, there are a lot of things that are squirreled away, and with little assistance.

I have one on ssh right now - they intercept certain sshd calls but I can’t see any much documentation.

I can ssh normally to my server, but NOT ssh git@myserver depending on the client. They have changed some sshd security settings deep down but you can’t find it easily.

Hey ho.

Hi all

I found this thread when I was looking for “ready to go” information about running gitlab behind an already existing apache2 server (running on ports 80 and 443 with SSL enabled - using let’s encrypt).

This apache2 server is used for a lot of things (so it cannot be removed and replaced by gitlab).
SSL enabled and forced: every http request is forced https by apache.

Here is the “ready to go” information!

First you have to enable apache2 “proxy” mods:

a2enmod proxy
a2enmod proxy_http
service apache2 restart

Then you have to create a config file for redirecting URL which begins with /gitlab to the real gitlab server (which will be installed on the same machine, in this example, listening on port 81)

/etc/apache2/conf-available/proxy-to-gitlab.conf:

ProxyRequests off
ProxyPass "/gitlab" "http://127.0.0.1:81/gitlab" retry=0
ProxyPassReverse "/gitlab" "http://127.0.0.1:81/gitlab"

Then enabling this configuration file by playing

a2enconf proxy-to-gitlab 
service apache2 restart

(this creates a link agains this file into conf-enabled folder, and starting from now, apache2 is ready).

In the “gitlab” side, after installation, into /etc/gitlab/gitlab.rb:

external_url 'http://real-apache2-url.com/gitlab'
gitlab_rails['trusted_proxies'] = ['127.0.0.1']
nginx['listen_addresses'] = ['127.0.0.1']
nginx['listen_port'] = 81
nginx['real_ip_trusted_addresses'] = ['127.0.0.1']
nginx['real_ip_header'] = 'X-Forwarded-For'
nginx['real_ip_recursive'] = 'on'

Mind the “http”: it’s much easier to avoid gitlab messing with https stuff since apache2 is already doing everything fine, in a transparent way so that gitlab doesn’t have to handle this.

After that, running “gitlab-ctl reconfigure” and everything worked fine!

Real user’s IP addresses are correctly seen, gitlab isn’t “https aware” but every http link is forced https by apache (even git clone commands) so that’s all perfect and of course, what was on the apache2 server is still working.

I fixed mine using this apache rewrite rule

    ProxyRequests Off
    SSLProxyEngine on
    SSLProxyVerify none 
    SSLProxyCheckPeerCN off
    SSLProxyCheckPeerName off
    SSLProxyCheckPeerExpire off
    ProxyPass / https://localhost:30003/
    ProxyPassReverse / https://gitlab.example.com/

Btw my gitlab is running on docker and reverse proxy by aapanel apache