Renweing SSL certs with custom NGINX config leading to errors

I’ve got a somewhat unusual configuration that I need some help with. I’ve written a script that uses the GitLab API to scrape one of our groups to generate a Satis build. This Satis build is then served through GitLab’s NGINX configuration. This is working great, except when I run sudo gitlab-ctl reconfigure, it fails to generate new Let’s Encrypt certificates because the file at /.well-known/acme-challenge doesn’t exist.

[2024-02-20T16:06:59-06:00] ERROR: Running exception handlers
There was an error running gitlab-ctl reconfigure:

letsencrypt_certificate[git.example.com] (letsencrypt::http_authorization line 6) had an error: RuntimeError: acme_certificate[staging] (letsencrypt::http_authorization line 43) had an error: RuntimeError: ruby_block[create certificate for git.example.com] (letsencrypt::http_authorization line 110) had an error: RuntimeError: [git.example.com] Validation failed, unable to request certificate, Errors: [{url: https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/11280587003/hFulEg, status: invalid, error: {"type"=>"urn:ietf:params:acme:error:unauthorized", "detail"=>"24.1.54.234: Invalid response from https://packages.example.com/.well-known/acme-challenge/aLPuEPizdYbNPiV3p0PJxls_a17yiKeLl0IJhwLFldQ: 404", "status"=>403}} ]

The Satis build is using user and group www-data. I’ve tried adding both gitlab-www and git to the www-data group, but neither of those seems to fix the issue.

What do I need to do to allow the cookbook to generate the challenge file?

What is a Satis build? Maybe you can show the script, and show how this is configured to be served through the GitLab Nginx host. I’m not sure I understand the architecture.

Satis is a system for hosting your own Composer repositories privately; I think the particular system being used is rather irrelevant, this would occur with any custom_nginx_config site as far as I can tell.

The domain is set up using the custom_nginx_config setting as described here NGINX settings | GitLab

The config file looks like this:

server {
    listen 80;
    listen [::]:80;
    server_name packages.example.com;

    return 301 https://packages.example.com$request_uri;
}

server {
    listen 443 ssl;
    listen [::]:443 ssl;
    server_name packages.example.com;

    ssl_certificate /etc/gitlab/ssl/packages.example.com.crt;
    ssl_certificate_key /etc/gitlab/ssl/packages.example.com.key;

    root /var/www/packages.example.com/html/web;
    index index.html index.htm index.nginx-debian.html;

    location / {
        try_files $uri $uri/ =404;
    }

    location /build {
        root /usr/lib/cgi-bin/;
        fastcgi_pass  unix:/var/run/fcgiwrap.socket;
        include /opt/gitlab/embedded/conf/fastcgi_params;
        fastcgi_param SCRIPT_FILENAME  /usr/lib/cgi-bin/satis.sh;
        fastcgi_read_timeout 600;
    }
}

The NGINX config itself is fine; I can visit the domain without issue, and if I manually create the .well-known/acme-challenge directory, I can access that via browser too, but GitLab just seems unable to write to this directory and I’m stumped as to why. It could be that it’s writing somewhere else, I’m just not sure where to even begin looking. This did work initially, which is why I’m so confused.

I can post more details about Satis if you want, but again, I think it’s largely irrelevant

Thanks for the details, learned something new with Satis builds myself. I agree, it should not be relevant, but it helps to draw the bigger picture of involved components :slight_smile:

https://packages.example.com/.well-known/acme-challenge/aLPuEPizdYbNPiV3p0PJxls_a17yiKeLl0IJhwLFldQ: 404

AFAIK the Nginx root directory is served from /var/opt/gitlab/nginx/www/ using the Omnibus packages. Maybe this location was modified and excludes the GitLab service user/group from writing into the directory. Depending on the host distribution, it may also be related to SELinux or AppArmor policies.

Another guess: Is /var/opt/gitlab mounted on an external disk, or remote cloud filesystem like Ceph?

Interesting, okay, so does that mean I need to move my root directory in to there?

I missed this configuration directive initially. It looks like the problem with serving the ACME challenge, because it is not in the root directory. Changing root to the GitLab location can be worth a try.

You could also try with a symlink but this would require that Nginx is allowed to follow symlinks. AFAIK this is a security setting which is disabled by default.

Tried moving it in to there, but I’m getting the same result :confused:

The file’s definitely not being created, at least not in the right place; I wrote this little script to watch for file creation in the directory, and it never generates. Do you know where these challenge files are supposed to be generated at? As you suggested, I’m wondering if I can just symlink it or something (I can adjust settings to allow that if needed).

#!/bin/bash

DIRECTORY_TO_OBSERVE="/var/www/packages.example.com/html/web/.well-known/acme-challenge"

echo "Starting to watch ${DIRECTORY_TO_OBSERVE} for changes"
inotifywait -m -r -e create --format '%w%f' "${DIRECTORY_TO_OBSERVE}" | while read NEW_FILE
do
        echo "A file has been created: ${NEW_FILE}"
done

Figured it out! I looked through the logs, and found the challenge file was being created at /var/opt/gitlab/nginx/www/.well-known, so I just updated my server config like so:

server {
    listen 80;
    listen [::]:80;
    server_name packages.example.com;

    location ^~ /.well-known {
        allow all;
        root /var/opt/gitlab/nginx/www;
        default_type "text/plain";
        try_files $uri =404;
    }

    return 301 https://packages.example.com$request_uri;
}

server {
    listen 443 ssl;
    listen [::]:443 ssl;
    server_name packages.example.com;

    ssl_certificate /etc/gitlab/ssl/packages.example.com.crt;
    ssl_certificate_key /etc/gitlab/ssl/packages.example.com.key;

    root /var/www/packages.example.com/html/web;
    index index.html index.htm index.nginx-debian.html;

    location ^~ /.well-known {
        allow all;
        root /var/opt/gitlab/nginx/www;
        default_type "text/plain";
        try_files $uri =404;
    }

    location / {
        try_files $uri $uri/ =404;
    }

    location /build {
        root /usr/lib/cgi-bin/;
        fastcgi_pass  unix:/var/run/fcgiwrap.socket;
        include /opt/gitlab/embedded/conf/fastcgi_params;
        fastcgi_param SCRIPT_FILENAME  /usr/lib/cgi-bin/satis.sh;
        fastcgi_read_timeout 600;
    }
}

Not necessarily the most elegant of solutions, could run in to issues if I ever need multiple custom domains, but hey, this works well enough for me.

1 Like