"error":"keywatcher: dial unix /var/opt/gitlab/redis/redis.socket

Downloaded that latest ubuntu version and installed on 20.04
Getting a 502 error when trying to connect via browser.

Logs show some socket errors:

{“level”:“info”,“msg”:“keywatcher: starting process loop”,“time”:“2021-05-04T15:36:45Z”}
{“address”:"/var/opt/gitlab/redis/redis.socket",“level”:“info”,“msg”:“redis: dialing”,“network”:“unix”,“time”:“2021-05-04T15:36:45Z”}
{“error”:“keywatcher: dial unix /var/opt/gitlab/redis/redis.socket: connect: no such file or directory”,“level”:“error”,“msg”:"",“time”:“2021-05-04T15:36:45Z”}
{“address”:"/var/opt/gitlab/redis/redis.socket",“level”:“info”,“msg”:“redis: dialing”,“network”:“unix”,“time”:“2021-05-04T15:36:45Z”}
{“error”:“keywatcher: dial unix /var/opt/gitlab/redis/redis.socket: connect: no such file or directory”,“level”:“error”,“msg”:"",“time”:“2021-05-04T15:36:45Z”}
{“address”:"/var/opt/gitlab/redis/redis.socket",“level”:“info”,“msg”:“redis: dialing”,“network”:“unix”,“time”:“2021-05-04T15:36:45Z”}
{“error”:“keywatcher: dial unix /var/opt/gitlab/redis/redis.socket: connect: no such file or directory”,“level”:“error”,“msg”:"",“time”:“2021-05-04T15:36:45Z”}
{“address”:"/var/opt/gitlab/redis/redis.socket",“level”:“info”,“msg”:“redis: dialing”,“network”:“unix”,“time”:“2021-05-04T15:36:45Z”}
{“error”:“keywatcher: dial unix /var/opt/gitlab/redis/redis.socket: connect: no such file or directory”,“level”:“error”,“msg”:"",“time”:“2021-05-04T15:36:45Z”}
{“address”:"/var/opt/gitlab/redis/redis.socket",“level”:“info”,“msg”:“redis: dialing”,“network”:“unix”,“time”:“2021-05-04T15:36:46Z”}
{“error”:“keywatcher: dial unix /var/opt/gitlab/redis/redis.socket: connect: no such file or directory”,“level”:“error”,“msg”:"",“time”:“2021-05-04T15:36:46Z”}
{“address”:"/var/opt/gitlab/redis/redis.socket",“level”:“info”,“msg”:“redis: dialing”,“network”:“unix”,“time”:“2021-05-04T15:36:47Z”}
{“correlation_id”:“01F4W0H08FY2VA0K2D0CNHY2F5”,“duration_ms”:0,“error”:“badgateway: failed to receive response: dial unix /var/opt/gitlab/gitlab-rails/sockets/gitlab.socket: connect: connection refused”,“level”:“error”,“method”:“GET”,“msg”:"",“time”:“2021-05-04T15:39:12Z”,“uri”:"/"}

Redis config has TCP port set to zero.

Hi @mthomasPRS
please provide output from gitlab-ctl status and any errors from /var/log/gitlab/redis/current

Hi!

Requested information below…

run: gitaly: (pid 693077) 9s; run: log: (pid 657922) 156512s
run: gitlab-workhorse: (pid 693092) 8s; run: log: (pid 658408) 156321s
run: logrotate: (pid 693103) 8s; run: log: (pid 657826) 156524s
run: nginx: (pid 693113) 7s; run: log: (pid 658431) 156313s
run: postgresql: (pid 693121) 7s; run: log: (pid 658070) 156501s
run: puma: (pid 693123) 6s; run: log: (pid 658329) 156334s
run: redis: (pid 693135) 6s; run: log: (pid 657864) 156518s
run: registry: (pid 693149) 5s; run: log: (pid 659103) 156154s
run: sidekiq: (pid 693151) 5s; run: log: (pid 658354) 156328s

2021-05-05_13:55:05.11293 693135:C 05 May 2021 13:55:05.112 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
2021-05-05_13:55:05.11299 693135:C 05 May 2021 13:55:05.112 # Redis version=6.0.12, bits=64, commit=f86385f7, modified=1, pid=693135, just started
2021-05-05_13:55:05.11300 693135:C 05 May 2021 13:55:05.112 # Configuration loaded
2021-05-05_13:55:05.13201 693135:M 05 May 2021 13:55:05.131 * Running mode=standalone, port=0.
2021-05-05_13:55:05.13204 693135:M 05 May 2021 13:55:05.131 # Server initialized
2021-05-05_13:55:05.13206 693135:M 05 May 2021 13:55:05.132 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add ‘vm.overcommit_memory = 1’ to /etc/sysctl.conf and then reboot or run the command ‘sysctl vm.overcommit_memory=1’ for this to take effect.
2021-05-05_13:55:05.16716 693135:M 05 May 2021 13:55:05.167 * Loading RDB produced by version 6.0.12
2021-05-05_13:55:05.16718 693135:M 05 May 2021 13:55:05.167 * RDB age 78263 seconds
2021-05-05_13:55:05.16719 693135:M 05 May 2021 13:55:05.167 * RDB memory usage when created 5.62 Mb
2021-05-05_13:55:05.17031 693135:M 05 May 2021 13:55:05.170 * DB loaded from disk: 0.037 seconds
2021-05-05_13:55:05.17032 693135:M 05 May 2021 13:55:05.170 * The server is now ready to accept connections at /var/opt/gitlab/redis/redis.socket

Perms:

-rw------- 1 gitlab-redis gitlab-redis 84955 May 4 16:10 dump.rdb
-rw-r–r-- 1 gitlab-redis root 52131 May 3 18:26 redis.conf
srwxrwxrwx 1 gitlab-redis gitlab-redis 0 May 5 13:55 redis.socket

Error 502 is also present during GitLab startup. It usually goes away after couple of minutes. If not make sure you meet the hardware requirements.

embedded redis is configured using the redis['*'] keys. If you are using the embedded Redis any gitlab_rails['redis_*'] keys shouldn’t be defined in gitlab.rb.

I have left it 15+ minutes and get the same error. It is a 12 core box with 5GB of free ram.

I am not a redis user, can you clarify what you mean? Do I need to make any redis changes?