Upgrading GitLab on Docker from 15.9.1 to 15.11.5 (or 16.0.1) : Issues with Redis?

Hello,

I am trying to upgrade GitLab from 15.9.1 to 15.11.5.
The upgrade process hangs, and I can see the following problem in the log:

==> /var/log/gitlab/gitlab-workhorse/current <==
{"error":"keywatcher: pubsub receive: EOF","level":"error","msg":"","time":"2023-05-21T10:18:52Z"}
{"address":"/var/opt/gitlab/redis/redis.socket","level":"info","msg":"redis: dialing","network":"unix","time":"2023-05-21T10:18:52Z"}

After restarting GitLab, I can see Redis coming up:
2023-05-21_10:17:51.56794 3112:M 21 May 2023 10:17:51.567 * The server is now ready to accept connections at /var/opt/gitlab/redis/redis.socket

Running gitlab-redis-cli --stat shows the following output:

------- data ------ --------------------- load -------------------- - child -                                                                                                           
keys       mem      clients blocked requests            connections                                                                                                                     
1762       4.45M    29      20      17804 (+0)          100                                                                                                                             
1762       4.26M    29      20      17830 (+26)         100

I also tried to modify the permissions on the redis.sock socket (since a previous error in the upgrade pointed to incorrect permissions on the .ssh/authorized_keys file (I changed the owner to git:git), so I have this now:

/var/opt/gitlab/redis# ls -al
...
srwxrwxrwx 1 git          git               0 May 21 10:09 redis.socket

I un-commented all redis[*] related lines in the gitlab.rb file:

redis['enable'] = true
redis['ha'] = false
redis['hz'] = 10
redis['dir'] = "/var/opt/gitlab/redis"
redis['log_directory'] = "/var/log/gitlab/redis"
redis['username'] = "gitlab-redis"
redis['group'] = "gitlab-redis"
redis['maxclients'] = "10000"
redis['maxmemory'] = "0"
redis['maxmemory_policy'] = "noeviction"
redis['maxmemory_samples'] = "5"
redis['tcp_backlog'] = 511
redis['tcp_timeout'] = "60"
redis['tcp_keepalive'] = "300"
redis['uid'] = nil
redis['gid'] = nil

But this didn’t help either…

Any ideas on what else I can try?

Many thanks in advance for your help and suggestions!

Update:
I failed to mention that my GitLab environment is running in Docker, on a Synology DSM OS.

I now tried to upgrade again to the latest docker image (presumably 16.0.1) but I get the same error message:

==> /var/log/gitlab/gitlab-workhorse/current <==                                                                                                                                                                                                                                         
{"error":"keywatcher: pubsub receive: EOF","level":"error","msg":"","time":"2023-05-25T14:47:17Z"}                                                                                                                                                                                       
{"address":"/var/opt/gitlab/redis/redis.socket","level":"info","msg":"redis: dialing","network":"unix","time":"2023-05-25T14:47:17Z"}

I tried to get some more information but I can’t find anything useful:

root@gitlab-ce-15:/# tail /var/log/gitlab/nginx/gitlab_access.log                                                                                                                                                                                                                        
127.0.0.1 - - [25/May/2023:14:47:02 +0000] "GET /help HTTP/1.1" 200 74516 "" "curl/8.0.1-DEV" -                                                                                                                                                                                          
127.0.0.1 - - [25/May/2023:14:48:03 +0000] "GET /help HTTP/1.1" 200 74516 "" "curl/8.0.1-DEV" -

and

root@gitlab-ce-15:/# tail /var/log/gitlab/redis/current                                                                                                                                                                                                                                  
2023-05-25_14:45:02.00792 274:M 25 May 2023 14:45:02.007 * 10 changes in 300 seconds. Saving...                                                                                                                                                                                          
2023-05-25_14:45:02.00827 274:M 25 May 2023 14:45:02.008 * Background saving started by pid 880                                                                                                                                                                                          
2023-05-25_14:45:02.02514 880:C 25 May 2023 14:45:02.025 * DB saved on disk                                                                                                                                                                                                              
2023-05-25_14:45:02.02611 880:C 25 May 2023 14:45:02.026 * RDB: 0 MB of memory used by copy-on-write                                                                                                                                                                                     
2023-05-25_14:45:02.10861 274:M 25 May 2023 14:45:02.108 * Background saving terminated with success                                                                                                                                                                                     
2023-05-25_14:50:03.05249 274:M 25 May 2023 14:50:03.052 * 10 changes in 300 seconds. Saving...                                                                                                                                                                                          
2023-05-25_14:50:03.05273 274:M 25 May 2023 14:50:03.052 * Background saving started by pid 995                                                                                                                                                                                          
2023-05-25_14:50:03.06302 995:C 25 May 2023 14:50:03.062 * DB saved on disk                                                                                                                                                                                                              
2023-05-25_14:50:03.06343 995:C 25 May 2023 14:50:03.063 * RDB: 0 MB of memory used by copy-on-write                                                                                                                                                                                     
2023-05-25_14:50:03.15311 274:M 25 May 2023 14:50:03.153 * Background saving terminated with success

Any ideas?

Thanks for your help!

Not entirely sure what docker version is on Synology, but this link shows some problems with upgrades to 16.x: Updating from 15.11 to latest docker build fails - #9 by paula.kokic

Did the upgrade finish successfully? What was the output from the upgrade logs? First we need to make sure the upgrade actually finished successfully before concentrating on Redis. Either way, unless docker is of the version in the link above, then you won’t be able to go to Gitlab 16 on Synology until Synology upgrade the docker version to a supported one.

You’ll need to check that though, as I don’t use docker on Synology.

Thanks for the hint. I think I am still using Docker 20.10.3:

Is Docker 20.10.10++ also needed for GitLab 15.11.x ? Because I originally tried to upgrade to that version and got stuck at the Redis issue.

I tried to look at the logs, here are the last lines from today’s upgrade attempt:

root@gitlab-ce-15:/var/log/gitlab/reconfigure# tail 1685025366.log                                                                                             
[2023-05-25T14:38:25+00:00] INFO: templatesymlink[Create a gitlab.yml and create a symlink to Rails root] sending run action to execute[clear the gitlab-rails 
cache] (delayed)                                                                                                                                               
[2023-05-25T14:38:58+00:00] INFO: execute[clear the gitlab-rails cache] ran successfully                                                                       
[2023-05-25T14:38:58+00:00] INFO: version_file[Create version file for Gitaly] sending hup action to runit_service[gitaly] (delayed)                           
[2023-05-25T14:38:58+00:00] INFO: runit_service[gitaly] signalled (HUP)                                                                                        
[2023-05-25T14:38:58+00:00] INFO: runit_service[gitaly] sent hup                                                                                               
[2023-05-25T14:38:58+00:00] INFO: version_file[Create version file for Gitlab KAS] sending restart action to runit_service[gitlab-kas] (delayed)               
[2023-05-25T14:39:09+00:00] INFO: env_dir[/opt/gitlab/etc/gitlab-workhorse/env] sending restart action to runit_service[gitlab-workhorse] (delayed)            
[2023-05-25T14:39:09+00:00] INFO: Cinc Client Run complete in 181.756971988 seconds                                                                            
[2023-05-25T14:39:09+00:00] INFO: Running report handlers                                                                                                      
[2023-05-25T14:39:09+00:00] INFO: Report handlers complete

And here are the last lines from the upgrade attempt to 15.11.5:

root@gitlab-ce-15:/var/log/gitlab/reconfigure# tail 1684656555.log -n 20                                                                                       
[2023-05-21T08:11:39+00:00] INFO: link[/opt/gitlab/init/grafana] created                                                                                       
[2023-05-21T08:11:39+00:00] INFO: template[/opt/gitlab/sv/grafana/run] sending run action to ruby_block[restart_service] (delayed)                             
[2023-05-21T08:11:39+00:00] INFO: template[/opt/gitlab/sv/grafana/log/config] sending create action to ruby_block[reload_log_service] (delayed)                
[2023-05-21T08:11:39+00:00] INFO: link[/opt/gitlab/service/grafana] created                                                                                    
[2023-05-21T08:11:43+00:00] INFO: ruby_block[wait for grafana service socket] called                                                                           
[2023-05-21T08:11:43+00:00] INFO: ruby_block[reload_log_service] called                                                                                        
[2023-05-21T08:11:43+00:00] INFO: template[/opt/gitlab/sv/grafana/log/run] sending run action to ruby_block[restart_log_service] (delayed)                     
[2023-05-21T08:11:44+00:00] INFO: ruby_block[restart_log_service] called                                                                                       
[2023-05-21T08:11:44+00:00] INFO: templatesymlink[Create a gitlab.yml and create a symlink to Rails root] sending run action to execute[clear the gitlab-rails 
cache] (delayed)                                                                                                                                               
[2023-05-21T08:12:19+00:00] INFO: execute[clear the gitlab-rails cache] ran successfully                                                                       
[2023-05-21T08:12:19+00:00] INFO: version_file[Create version file for Gitaly] sending hup action to runit_service[gitaly] (delayed)                           
[2023-05-21T08:12:19+00:00] INFO: runit_service[gitaly] signalled (HUP)                                                                                        
[2023-05-21T08:12:19+00:00] INFO: runit_service[gitaly] sent hup                                                                                               
[2023-05-21T08:12:19+00:00] INFO: version_file[Create version file for Gitlab KAS] sending restart action to runit_service[gitlab-kas] (delayed)               
[2023-05-21T08:12:29+00:00] INFO: env_dir[/opt/gitlab/etc/gitlab-workhorse/env] sending restart action to runit_service[gitlab-workhorse] (delayed)            
[2023-05-21T08:12:29+00:00] INFO: env_dir[/opt/gitlab/etc/grafana/env] sending restart action to runit_service[grafana] (delayed)                              
[2023-05-21T08:12:30+00:00] INFO: Cinc Client Run complete in 191.179007107 seconds                                                                            
[2023-05-21T08:12:30+00:00] INFO: Running report handlers                                                                                                      
[2023-05-21T08:12:30+00:00] INFO: Report handlers complete                                                                                                     
[2023-05-21T08:12:30+00:00] WARN: This release of Cinc Client became end of life (EOL) on May 1st 2023. Please update to a supported release to receive new fea
tures, bug fixes, and security updates.

If I try to look for “ERROR” inside these log files, there are no results.

Any ideas?
Many tanks for the help!

I’ve updated my Synology box to DSM 7.2. Now, the Docker version should be 20.10.23:

I’ve also updated the GitLab container image to 16.0.4, reset the container and tried again.

I’m still getting the same error message.
What can I do to get this up?
Any help is much appreciated!

Did the following:

  1. Created a new container with GitLab 16.1.1
  2. Ran the new (empty) container - started with no issues on my Synology host
  3. Shut down the new container
  4. Edited the container settings and mapped the config + data + logs folders to my “production” GitLab container’s directories
  5. Started the container again. GitLab is up and running !

I can see some error messages in the logs, but

gitlab-rake gitlab:check

returned no issues.