Unresponsive home page with only header showing (see the picture in attachment)

I’m running the GitLab server on Ubuntu 18.04 and after the initial password change, I’ve got the home screen but only with a header and in a non-response state. You can see it in the picture below.

I would be very grateful if someone can give me a guide on how to find and solve this issue. I’ve followed all the instructions from the GitLab website.

Hi,

Please check to ensure all services are running:

gitlab-ctl status

perhaps even reconfigure and restart and try again:

gitlab-ctl reconfigure
gitlab-ctl restart

as this sometimes helps when problems occur and is best place to start.

Hi,
Thank you for the help.

I’ve sent all the commands you mentioned and I’m getting the same results. You can see the output from gitlab-ctl status command below:

gitlab-ctl status
run: alertmanager: (pid 27069) 20s; run: log: (pid 23734) 86329s
run: gitaly: (pid 27083) 18s; run: log: (pid 20604) 87212s
run: gitlab-exporter: (pid 27107) 18s; run: log: (pid 23612) 86343s
run: gitlab-workhorse: (pid 27115) 17s; run: log: (pid 21676) 86807s
run: grafana: (pid 27132) 17s; run: log: (pid 24143) 86236s
run: logrotate: (pid 27148) 16s; run: log: (pid 20517) 87224s
run: node-exporter: (pid 27154) 16s; run: log: (pid 23553) 86347s
run: postgres-exporter: (pid 27160) 15s; run: log: (pid 23763) 86326s
run: postgresql: (pid 27212) 15s; run: log: (pid 20761) 87194s
run: prometheus: (pid 27252) 14s; run: log: (pid 23684) 86334s
run: puma: (pid 27278) 13s; run: log: (pid 21564) 86823s
run: redis: (pid 27283) 12s; run: log: (pid 20551) 87218s
run: redis-exporter: (pid 27289) 11s; run: log: (pid 23634) 86340s
run: sidekiq: (pid 27303) 8s; run: log: (pid 21595) 86817s

You will need to go through all the log files under /var/log/gitlab and see if you can find any errors. Alternatively, delete the install completely and try again. Maybe your install is corrupt.

I did look up logs, but nothing suspicious was there. Will look it up again.

Can you send me the instructions how to completely remove gitlab from my system - Ubuntu 18.04?

Depending on whether you installed gitlab-ce or gitlab-ee, then it would be:

apt purge gitlab-ce

or change ce to ee. Then make sure /opt/gitlab has gone, as well as /var/opt/gitlab and /etc/gitlab. Then make a clean install.

I suggest doing:

apt-get clean all

to delete any previously downloaded packages, to make sure there are new fresh downloads.

Thank you. Will get back with the results.

I’ve done the complete reinstall and the same problem is there.

I found this error:

Failed opening the RDB file dump.rdb (in server root dir unknown) for saving: No such file or directory

Can it be the reason for the error?

Check permissions on /var/opt/gitlab/redis/dump.rdb it should look like mine:

ls -lha /var/opt/gitlab/redis/dump.rdb 
-rw------- 1 gitlab-redis gitlab-redis 556K May 12 15:51 /var/opt/gitlab/redis/dump.rdb

if it doesn’t match this, then:

chown gitlab-redis:gitlab-redis /var/opt/gitlab/redis/dump.rdb

and then restart gitlab:

gitlab-ctl restart

Check redis is running:

netstat -tunlp | grep -i redis
tcp        0      0 127.0.0.1:9121          0.0.0.0:*               LISTEN      3779/redis_exporter 

you can also do this to check the processes are running:

ps aux | grep -i redis | grep -i gitlab
gitlab-+  2773  1.7  0.0  75708 12976 ?        Ssl  Apr28 361:44 /opt/gitlab/embedded/bin/redis-server 127.0.0.1:0
gitlab-+  3779  0.0  0.0 1246516 22760 ?       Ssl  May04   7:11 /opt/gitlab/embedded/bin/redis_exporter --web.listen-address=localhost:9121 --redis.addr=unix:///var/opt/gitlab/redis/redis.socket
root     16914  0.0  0.0   2304  1256 ?        S     2020   0:01 svlogd -tt /var/log/gitlab/redis-exporter
root     16917  0.0  0.0   2304  1240 ?        S     2020   0:27 svlogd -tt /var/log/gitlab/redis

it should also show as running under:

gitlab-ctl status

as well.

Permissions are fine:

-rw------- 1 gitlab-redis gitlab-redis 244K May 13 02:32 /var/opt/gitlab/redis/dump.rdb

Redis is running but the port is different(31494) than yours and it says redis_exporte instead of redis_exporter

netstat -tunlp | grep -i redis
tcp        0      0 127.0.0.1:9121          0.0.0.0:*               LISTEN      31494/redis_exporte

There is some errors with the following command (please scroll to the right to see the error):

ps aux | grep -i redis | grep -i gitlab
root     20472  0.0  0.0   4556  1312 ?        Ss   May11   0:18 runsvdir -P /opt/gitlab/service log: ies. svlogd: warning: unable to lock directory: /var/log/gitlab/redis: temporary failure svlogd: fatal: no functional log directories. svlogd: warning: unable to lock directory: /var/log/gitlab/redis: temporary failure svlogd: fatal: no functional log directories. svlogd: warning: unable to lock directory: /var/log/gitlab/redis: temporary failure svlogd: fatal: no functional log directories.
root     20551  0.0  0.0   4548   836 ?        S    May11   0:03 svlogd -tt /var/log/gitlab/redis
root     25808  0.0  0.0   4548   804 ?        S    May12   0:00 svlogd -tt /var/log/gitlab/redis-exporter
gitlab-+ 27283  0.5  0.0  81132  8144 ?        Ssl  May12   4:55 /opt/gitlab/embedded/bin/redis-server 127.0.0.1:0
gitlab-+ 31486  4.2  0.0  86252 12528 ?        Ssl  May12  33:06 /opt/gitlab/embedded/bin/redis-server 127.0.0.1:0
gitlab-+ 31494  0.1  0.1 1554692 24008 ?       Ssl  May12   1:32 /opt/gitlab/embedded/bin/redis_exporter --web.listen-address=localhost:9121 --redis.addr=unix:///var/opt/gitlab/redis/redis.socket

And as a result redis is down:

gitlab-ctl status
run: alertmanager: (pid 31262) 47406s; run: log: (pid 25918) 48727s
run: gitaly: (pid 31276) 47405s; run: log: (pid 21302) 50395s
run: gitlab-exporter: (pid 31298) 47405s; run: log: (pid 25763) 48763s
run: gitlab-workhorse: (pid 31302) 47405s; run: log: (pid 25697) 48773s
run: grafana: (pid 31316) 47404s; run: log: (pid 26492) 48625s
run: logrotate: (pid 11674) 596s; run: log: (pid 24079) 49659s
run: node-exporter: (pid 31348) 47403s; run: log: (pid 25744) 48768s
run: postgres-exporter: (pid 31359) 47403s; run: log: (pid 25947) 48720s
run: postgresql: (pid 31373) 47402s; run: log: (pid 21592) 50338s
run: prometheus: (pid 31375) 47401s; run: log: (pid 25855) 48737s
run: puma: (pid 31476) 47401s; run: log: (pid 25609) 48785s
run: redis: (pid 31486) 47399s; down: log: 0s, normally up, want up
run: redis-exporter: (pid 31494) 47398s; run: log: (pid 25808) 48751s
run: sidekiq: (pid 31510) 47395s; run: log: (pid 25642) 48779s

Make sure that redis directory exists under /var/log/gitlab.

31494 is the process id not port. This will always change and each restart. The port is 9121.

Redis directory exists:

ls -lh /var/log/gitlab
total 80K
drwx------ 2 gitlab-prometheus root       4.0K May 12 10:53 alertmanager
drwx------ 2 git               root       4.0K May 13 00:39 gitaly
drwx------ 2 git               root       4.0K May 12 10:52 gitlab-exporter
drwx------ 2 git               root       4.0K May 13 03:30 gitlab-rails
drwx------ 2 git               root       4.0K May 11 10:37 gitlab-shell
drwx------ 2 git               root       4.0K May 12 10:45 gitlab-workhorse
drwx------ 2 gitlab-prometheus root       4.0K May 12 10:54 grafana
drwxr-xr-x 2 root              root       4.0K May 11 10:45 lets-encrypt
drwx------ 2 root              root       4.0K May 12 10:38 logrotate
drwxr-x--- 2 root              gitlab-www 4.0K May 12 00:26 nginx
drwx------ 2 gitlab-prometheus root       4.0K May 12 10:52 node-exporter
drwx------ 2 gitlab-psql       root       4.0K May 12 10:53 postgres-exporter
drwx------ 2 gitlab-psql       root       4.0K May 12 10:38 postgresql
drwx------ 2 gitlab-prometheus root       4.0K May 12 10:53 prometheus
drwx------ 2 git               root       4.0K May 13 00:39 puma
drwxr-xr-x 2 root              root       4.0K May 12 12:47 reconfigure
drwx------ 2 gitlab-redis      root       4.0K May 13 10:38 redis
drwx------ 2 gitlab-redis      root       4.0K May 12 10:53 redis-exporter
drwx------ 2 registry          root       4.0K May 11 10:45 registry
drwx------ 2 git               root       4.0K May 12 10:45 sidekiq

I posted wrong info for log directory, removed it so as not to confuse.

I think they are different because you’ve looked at the different paths when comparing to my results. My log was from /var/log/gitlab and you were telling me to set up the /var/opt/gitlab/redis.

Now after I’ve made the change I’ve new error:
{"error":"keywatcher: dial unix /var/opt/gitlab/redis/redis.socket: connect: permission denied","level":"error","msg":"","time":"2021-05-13T12:39:34+02:00"}

Can you send me the logs for those two directories above from your system?

My mistake, it should have been /var/log/gitlab/redis in the last post. Edited it to show correct path.

root@repo:# ls -lh  /var/opt/gitlab | grep redis
drwxr-x--- 2 gitlab-redis      git        4.0K May 13 14:30 redis

root@repo:# ls -lh /var/log/gitlab| grep redis
drwx------ 2 gitlab-redis      root       4.0K May 13 08:34 redis
drwx------ 2 gitlab-redis      root       4.0K May  5 08:35 redis-exporter

that’s how the permissions should look for both locations.

So to correct:

chown -R gitlab-redis:git /var/opt/gitlab/redis
chown -R gitlab-redis:root /var/log/gitlab/redis
chown -R gitlab-redis:root /var/log/gitlab/redis-exporter

then restart, if redis still doesn’t come up, then we need to find out why from your log files.

I set the commands you posted but got the same result, redis is not starting.
Here is the log from the redis service:

sudo gitlab-ctl tail redis
==> /var/log/gitlab/redis/state <==

==> /var/log/gitlab/redis/current <==
2021-05-13_20:55:56.05615 14502:C 13 May 2021 22:55:56.056 # Failed opening the RDB file dump.rdb (in server root dir unknown) for saving: No such file or directory
2021-05-13_20:55:56.15633 27283:M 13 May 2021 22:55:56.156 # Background saving error
2021-05-13_20:56:02.07864 27283:M 13 May 2021 22:56:02.078 * 1 changes in 900 seconds. Saving...
2021-05-13_20:56:02.07943 27283:M 13 May 2021 22:56:02.079 * Background saving started by pid 14522
2021-05-13_20:56:02.07986 14522:C 13 May 2021 22:56:02.079 # Failed opening the RDB file dump.rdb (in server root dir unknown) for saving: No such file or directory
2021-05-13_20:56:02.18005 27283:M 13 May 2021 22:56:02.179 # Background saving error
2021-05-13_20:56:08.00092 27283:M 13 May 2021 22:56:08.000 * 1 changes in 900 seconds. Saving...
2021-05-13_20:56:08.00136 27283:M 13 May 2021 22:56:08.001 * Background saving started by pid 14546
2021-05-13_20:56:08.00192 14546:C 13 May 2021 22:56:08.001 # Failed opening the RDB file dump.rdb (in server root dir unknown) for saving: No such file or directory
2021-05-13_20:56:08.10193 27283:M 13 May 2021 22:56:08.101 # Background saving error
2021-05-13_20:56:13.01945 27283:M 13 May 2021 22:56:13.019 * 1 changes in 900 seconds. Saving...
2021-05-13_20:56:13.02026 27283:M 13 May 2021 22:56:13.020 * Background saving started by pid 14571
2021-05-13_20:56:13.02066 14571:C 13 May 2021 22:56:13.020 # Failed opening the RDB file dump.rdb (in server root dir unknown) for saving: No such file or directory
2021-05-13_20:56:13.12067 27283:M 13 May 2021 22:56:13.120 # Background saving error
2021-05-13_20:56:19.04271 27283:M 13 May 2021 22:56:19.042 * 1 changes in 900 seconds. Saving...
2021-05-13_20:56:19.04320 27283:M 13 May 2021 22:56:19.043 * Background saving started by pid 14579
2021-05-13_20:56:19.04366 14579:C 13 May 2021 22:56:19.043 # Failed opening the RDB file dump.rdb (in server root dir unknown) for saving: No such file or directory
2021-05-13_20:56:19.14402 27283:M 13 May 2021 22:56:19.143 # Background saving error
2021-05-13_20:56:25.06558 27283:M 13 May 2021 22:56:25.065 * 1 changes in 900 seconds. Saving...
2021-05-13_20:56:25.06643 27283:M 13 May 2021 22:56:25.066 * Background saving started by pid 14599
2021-05-13_20:56:25.06686 14599:C 13 May 2021 22:56:25.066 # Failed opening the RDB file dump.rdb (in server root dir unknown) for saving: No such file or directory
2021-05-13_20:56:25.16697 27283:M 13 May 2021 22:56:25.166 # Background saving error

Seems they have gitlab-redis:git for the /var/opt/gitlab/redis directory, but then files inside have something else, we need to reset permissions on dump.rdb now.

chown gitlab-redis:gitlab-redis /var/opt/gitlab/redis/dump.rdb

although this shouldn’t happen. I’ve made many clean installs on Debian and never had this problem that you are having.

Having the same issue again. Should I try to reinstall the complete Gitlab again?