Have you experienced repeated gitlab 502 errors?

How to Use GitLab Upgrade
1

hello. I am posting this to ask about your experiences.
Currently, I am installing and operating GitLab 16.7.0 with docker compose in a closed network, self-managed manner.

However, after upgrading GitLab and GitLab Runner from 16.4.0 to 16.7.0, a 502 error occurs if usage increases even slightly. but i can still login and see the jobs.

After searching on the Internet, in the case of self-managed method, the GitLab representative changed the default port, 80 or 443, to a custom port, such as 9980 or 8143, but did not set the port conflict or change port to a sub-service within Gitlab such as puma, resulting in 502. We confirmed a case where .

However, our team used 16.4.0 gitlab and 16.7.0 gitlab in package form through dockerhub. so, we just use package as it is.

Above all, this 502 error did not occur in 16.4.0. When it is the same server and the same number of users.

Now, architect has questions about GitLab version 16.7.0. In this regard, I am curious whether you frequently experience error 502 while using 16.7.0 GitLab.

Meanwhile, while upgrading the GitLab version, TLS cert (SSL cert) was additionally applied. The relevant docker compose yaml is below: I would appreciate it if you could let me know what you know about this.

For reference, I know that it is customary to use port 443 for https, but due to our closed network policy, 443 cannot be used, and a custom port such like 5580 has been used. Please keep this in mind. Also, to avoid complications, I displayed the existing information by hiding it in <> form.

version: '3.8'
services:
  gitlab:
    image: ${IMAGE}
    restart: unless-stopped
    hostname: 'gitlab.<custom-domain-name>'
    extra_hosts:
      - "nexus.<custom-domain-name>:<custom-ip>"
    container_name: gitlab
    labels:
      name: gitlab
    environment:
      GITLAB_OMNIBUS_CONFIG: |
        external_url 'https://gitlab.<custom-domain-name>:5580'

        letsencrypt['enable']           = false
        nginx['redirect_http_to_https'] = true
        nginx['ssl_certificate']        = "/etc/gitlab/ssl/<pub-tls-company>.crt"
        nginx['ssl_certificate_key']    = "/etc/gitlab/ssl/<pub-tls-company>.key"
      TZ:
        <custom-timezone>
    ports:
      - '80:80'
      - '5580:5580'
      - '5522:22'
    volumes:
      - '${GITLAB_HOME}/config:/etc/gitlab'
      - '${GITLAB_HOME}/logs:/var/log/gitlab'
      - '${GITLAB_HOME}/data:/var/opt/gitlab'
    shm_size: '256m'
    networks:
      - <custom-network>
networks:
  <custom-network>:
    name: <custom-network>
    external: true
2

Http 5xx errors are server errors. Suggest inspecting the production log, and correlate the timestamps to better debug what’s wrong.

4

thank you for your contribution!

luckily, Our GitLab no longer responds with a 502 error.

Our team removed SSL because the requirements were unclear.
we also upgraded the GitLab version and made some adjustments to other settings.

I personally looked at the production logs and analyzed the cause according to your guide, but in the current team situation, it would be better not to comment further on my analysis.

Analyzing GitLab from a server perspective this time, it was a great pleasure for me to be able to examine the components of a truly complex, but in other words, artisanal solution. Thank you, and best of luck to the GitLab team.

6

We were having a very similar problem to this, explained in this issue.

The problem was this line in our gitlab.rb file.

puma['worker_processes'] = 0

Simply removing it (reverting to the default value) solved our crashes!

1 Like