How to solve "Whoops, GitLab is taking too much time to respond."

General
1

Hello, I have GitLab in my own server working for years. This week a problem started to happen.

When I load “https://gitlab.myserver.com”, immediately this error is shown: “Whoops, GitLab is taking too much time to respond.”

First, that message appears immediately, so there is not too much time to respond.

Second, normally, this was solved when I run “gitlab-ctl reconfigure”, however, today, that solution did not work either,

How can I solve that definetely? other posts concerning this same error does not help.

Most of them talk about that error when GitLab was just installed or it was just upgraded. This is not my case.

This is the result of “gitlab-ctl status”:

[root@node4037 ~]# gitlab-ctl status
down: alertmanager: 0s, normally up, want up; run: log: (pid 2041) 77911s
run: crond: (pid 2050) 77911s; run: log: (pid 2034) 77911s
run: gitaly: (pid 2052) 77911s; run: log: (pid 2036) 77911s
run: gitlab-exporter: (pid 13728) 14646s; run: log: (pid 2035) 77911s
run: gitlab-workhorse: (pid 10352) 530s; run: log: (pid 2040) 77912s
run: grafana: (pid 2055) 77912s; run: log: (pid 2043) 77912s
run: logrotate: (pid 24717) 2306s; run: log: (pid 2070) 77908s
run: nginx: (pid 2056) 77912s; run: log: (pid 2038) 77912s
run: node-exporter: (pid 2091) 77908s; run: log: (pid 2069) 77908s
run: postgres-exporter: (pid 2044) 77912s; run: log: (pid 2039) 77912s
run: postgresql: (pid 2046) 77912s; run: log: (pid 2032) 77912s
run: prometheus: (pid 2047) 77912s; run: log: (pid 2030) 77912s
run: puma: (pid 28963) 1s; run: log: (pid 2033) 77912s
run: redis: (pid 2051) 77912s; run: log: (pid 2037) 77912s
run: redis-exporter: (pid 2045) 77912s; run: log: (pid 2031) 77912s
run: sidekiq: (pid 28904) 2s; run: log: (pid 2042) 77912s

Not even server reboot solved the problem.

How can I solve this? Version is: gitlab-ce-13.5.4-ce.0.el7.x86_64

Thanks
Jaime

2

Check for processes that are using 100% CPU, considering that your gitlab version is old, I expect it has been compromised by the RCE vulnerability which leaves cryptocurrency miners running on your machine.

3

Hello, I have applied the hotpatch shown in that post but it is still not working.

I tried to upgrade to 13.8.8 version using yum install gitlab-ce-13.8.8 but an error was shown at the final step:

gitlab preinstall:
gitlab preinstall: Database backup failed! If you want to skip this backup, run the following command and try again:
gitlab preinstall:
gitlab preinstall:  sudo touch /etc/gitlab/skip-auto-backup
gitlab preinstall:
error: %pre(gitlab-ce-13.8.8-ce.0.el7.x86_64) scriptlet failed, exit status 1
Error in PREIN scriptlet in rpm package gitlab-ce-13.8.8-ce.0.el7.x86_64
  Verifying  : gitlab-ce-13.8.8-ce.0.el7.x86_64                                                                                                  1/2
gitlab-ce-13.5.4-ce.0.el7.x86_64 was supposed to be removed but is not!
  Verifying  : gitlab-ce-13.5.4-ce.0.el7.x86_64                                                                                                  2/2

Failed:
  gitlab-ce.x86_64 0:13.5.4-ce.0.el7                                        gitlab-ce.x86_64 0:13.8.8-ce.0.el7

Before proceeding with upgrade, I have run this comand to back up the database: gitlab-backup create, After a few seconds, this happened:

/usr/bin/gitlab-backup: line 69: 27899 Killed                  /opt/gitlab/bin/gitlab-rake gitlab:backup:create ${@}

What else can you suggest?

Thanks,
Jaime

4

I cannot load gitlab page anymore. It always throws a 502 error, even when I have applied the Hotpatch, replaced the exiftool, reconfigured, stopped and started the gitlab service, rebooted the computer, nothing works!

5

When seeing gitlab_error.log file under /var/opt/gitlab/nginx folder, this error appears when 502 error occurs:

2022/04/22 15:31:36 [crit] 32312#0: *25 connect() to unix:/var/opt/gitlab/gitlab-workhorse/sockets/socket failed (2: No such file or directory) while connecting to upstream, client: 45.11.22.33, server: gitlab.example.com, request: "GET / HTTP/1.1", upstream: "http://unix:/var/opt/gitlab/gitlab-workhorse/sockets/socket:/", host: "gitlab.example.com"

I listed /var/opt/gitlab/gitlab-workhorse folder but it is empty.

These are the folder permissions:

drwxr-x--- 2 git               gitlab-www 4096 Apr 22 14:03 gitlab-workhorse

What’s wrong with that?

6

Finally, I could update to latest version of GitLab but the same error persisted.

I thought first that the problem was due to alertmanager service not starting up. I solved that but the error continued.

At last, I fixed the problem using the solution from this post: /var/opt/gitlab/gitlab-rails/sockets/gitlab.socket: connect: connection refused" (#5706) · Issues · GitLab.org / omnibus-gitlab · GitLab

Puma was not enabled. Now I could continue using GitLab after I enabled it.

Regards
Jaime