Issue with GitLab Runner: Failing to Execute Jobs - SSL and Network Troubleshooting

GitLab CI/CD
,
1

Describe your question in as much detail as possible:

I’m experiencing an SSL certificate error when my GitLab Runner tries to execute jobs, despite being able to curl, nslookup, and access the web gui securely. Other network tests like ping and curl are successful, but the Runner fails with an SSL error specifically when accessing the Git repository. To clarify, all other functions on GitLab are working, I can access it normally from the browser with no issues and the certificate is valid with the correct fqdn listed in the cert.

Environment Description:

  • Local DNS: Points to HAProxy 10.0.0.115 for gitlab.redacted.com.
  • Reverse HAProxy Configuration: SSL passthrough with SNI inspection. Routes gitlab.redacted.com to GitLab-ce on 10.0.0.170:8443 for HTTPS and 10.0.0.170:8888 for HTTP.
  • GitLab-CE and Runner Setup: Both running in rootless Podman containers on a RHEL9 VM.
  • Network Tests: Ping, curl, git clone (with access token and credential authentication) tests to gitlab.redacted.com from gitlab runner container are all successful.

Versions:

  • GitLab CE Version: 16.6.0 (self-managed)
  • GitLab Runner Version: 16.6.0

What are you seeing, and how does that differ from what you expect to see?

  • Error Observed: The GitLab Runner fails with the following SSL error during job execution:
fatal: unable to access 'https://gitlab.redacted.com/chris/test.git/': SSL: no alternative certificate subject name matches target host name 'gitlab.redacted.com'

CI Configuration (.gitlab-ci.yml):

build:
  script:
    - echo "Hello World"

Full Error Message:

Using docker image sha256:e63b2e354f3427c477295d20443d51acc483231948afa094164a4627cd8c2d2d for quay.io/podman/stable:latest with digest quay.io/podman/stable@sha256:63f39dcf1e039bc7728af57af2e2475404a4e49c71727cf805121e6890af2f8e ...
Preparing environment
Running on runner-rxca5trwv-project-1-concurrent-0 via b6a04a199751...
Getting source from Git repository
Fetching changes with git depth set to 20...
Reinitialized existing Git repository in /builds/chris/test/.git/
fatal: unable to access 'https://gitlab.redacted.com/chris/test.git/': SSL: no alternative certificate subject name matches target host name 'gitlab.redacted.com'
Cleaning up project directory and file based variables
ERROR: Job failed: exit code 1
  • Expected Behavior: Successful cloning of the repository and execution of jobs by the GitLab Runner without SSL errors.

Troubleshooting Steps Taken:

  1. Ensured HAProxy is correctly configured for SSL passthrough.
  2. Checked network connectivity and ruled out firewall issues.
  3. Updated both GitLab Runner and GitLab server to the latest versions.
  4. Successfully performed direct git clone operations on the runner container, as well as curl.
  5. Reviewed Let’s Encrypt SSL certificate settings and configurations.
  6. Reviewed podman logs, HAProxy logs, with no anomaly’s noted.

Request for Help:

I am seeking advice on resolving the SSL certificate error that occurs when the GitLab Runner attempts to access the Git repository. Any insights or suggested solutions to address this SSL error and ensure successful job execution would be highly appreciated.

gitlab.rb Configuration

  • I’ve tried the following items in my gitlab.rb file, but they didn’t change anything:
# External URL should be set to the domain name that HAProxy listens for
external_url 'https://gitlab.redacted.com'

# Add the HAProxy IP to the list of trusted proxies
gitlab_rails['trusted_proxies'] = ['10.0.0.115']

# If you are using GitLab's bundled Nginx and it's handling SSL termination
nginx['listen_port'] = 80
nginx['listen_https'] = true

# If Let's Encrypt is enabled for SSL certificates
letsencrypt['enable'] = true

HAProxy Configuration:

# Frontend for HTTP
frontend http_front
    bind *:80
    acl is_gitlab hdr_beg(host) -i gitlab.redacted.com
    use_backend gitlab_http_backend if is_gitlab

# Frontend for HTTPS
frontend https_front
    bind *:443
    mode tcp
    tcp-request inspect-delay 5s
    tcp-request content accept if { req_ssl_hello_type 1 }
    acl is_gitlab req.ssl_sni -i gitlab.redacted.com
    use_backend gitlab_https_backend if is_gitlab

# Backend for HTTP traffic to GitLab
backend gitlab_http_backend
    server gitlab_http 10.0.0.170:8888 check

# Backend for HTTPS traffic to GitLab
backend gitlab_https_backend
    mode tcp
    server gitlab_https 10.0.0.170:8443 check
2

I also tested specifically adding the local DNS host to the podman network, and still no resolution. Any help would be appreciated.