Docker container with gitlab version >=16.7.0 can not be started due ruby error with urandom on Synology

ns1045 · January 21, 2024, 11:01am

When I want to run any gitlab-ce docker image with versions > 16.7.0 I get an error that container could not be started. Ruby can not use securerandom due file access problems of dev/urandom. similar to https://bugs.ruby-lang.org/issues/14716.
https://forum.gitlab.com/t/upgrade-from-16-6-2-ce-to-16-7-0-ce-fails/98104

/opt/gitlab/embedded/lib/ruby/3.1.0/securerandom.rb:75:in `urandom': failed to get urandom (RuntimeError)
	from /opt/gitlab/embedded/lib/ruby/3.1.0/securerandom.rb:75:in `singleton class'
	from /opt/gitlab/embedded/lib/ruby/3.1.0/securerandom.rb:42:in `<module:SecureRandom>'
	from /opt/gitlab/embedded/lib/ruby/3.1.0/securerandom.rb:41:in `<top (required)>'
	from <internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:149:in `require'
	from <internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:149:in `require'
	from /opt/gitlab/embedded/cookbooks/package/libraries/settings_dsl.rb:21:in `<top (required)>'
	from /opt/gitlab/embedded/cookbooks/package/libraries/deprecations.rb:3:in `require_relative'
	from /opt/gitlab/embedded/cookbooks/package/libraries/deprecations.rb:3:in `<top (required)>'
	from <internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:86:in `require'
	from <internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:86:in `require'
	from /opt/gitlab/embedded/service/omnibus-ctl/check_config.rb:18:in `load_file'
	from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/omnibus-ctl-0.6.0/lib/omnibus-ctl.rb:192:in `eval'
	from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/omnibus-ctl-0.6.0/lib/omnibus-ctl.rb:192:in `load_file'
	from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/omnibus-ctl-0.6.0/lib/omnibus-ctl.rb:187:in `block in load_files'
	from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/omnibus-ctl-0.6.0/lib/omnibus-ctl.rb:186:in `each'
	from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/omnibus-ctl-0.6.0/lib/omnibus-ctl.rb:186:in `load_files'
	from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/omnibus-ctl-0.6.0/bin/omnibus-ctl:29:in `<top (required)>'
	from /opt/gitlab/embedded/bin/omnibus-ctl:25:in `load'
	from /opt/gitlab/embedded/bin/omnibus-ctl:25:in `<main>'

With 16.6.4 and 16.6.5 the container starts up.
Docker version: 20.10.3.1308 (due preinstalled on synology NAS)
System: 3.10.108 #42962 SMP Mon May 29 14:35:41 CST 2023 x86_64 GNU/Linux synology_braswell_916+
container imager: gitlab/gitlab-ce:16.7.0-ce.0 and gitlab/gitlab-ce:16.8.0-ce.0

amuecke · January 25, 2024, 10:35am

I have the same issue on my Synology NAS (DS916+) all patches/updates installed:
Docker version 20.10.3, build 55f0773
Linux version 3.10.108 #42962 SMP Mon May 29 14:35:41 CST 2023

With [16.6.5-ee.0] I was able to start GitLab without this issue.

MrSchizo · January 26, 2024, 10:41am

Same here and still using DSM 7.1s docker. I guess the world is really pushing into that 7.2 update because with 7.2s container everything runs smoothly

ns1045 · January 26, 2024, 4:34pm

Unfortunately an update to 7.2 did not resolved the issue. The kernel remains on 3.10.*
So we must hope, that synology updates the kernel version soon

Random79 · January 29, 2024, 6:21pm

Same issue here.
One 2021 Synology model updated to DSM DSM 7.2.1-69057 Update 4 ( did the update fine
but two older 2016 models that had to be manually updatded to DSM 7.2.1-69057 Update 4 show the same error as above with 16.7.0 and “latest” gitlab-ce Docker images.
16.6.6 is the newest that works.

Random79 · February 23, 2024, 2:10pm

This has become critical.
There is a security release of 16.9.1/16.8.3/16.7.6 that doesn’t have a 16.6 release like the 2 security issues before.
So now we have to update but the update doesn’t run because the Kernel version stays at 3.10.108 .

iwalker · February 23, 2024, 2:56pm

Have you opened a support issue with Synology? Since Gitlab works in Docker pretty much on every Linux distro, and yet the problems here are only related to running it on Synology - it would seem that Synology is the problem here in something that they have excluded for some bizarre reason. I expect pretty much any other container app that works in a similar way like Gitlab will also fail on Synology due to something that has been omitted by Synology.

Also this issue: /usr/local/lib/ruby/3.1.0/securerandom.rb:75:in `urandom': failed to get urandom (RuntimeError) · Issue #383 · docker-library/ruby · GitHub suggests requiring a newer kernel as already mentioned - which you will have to ask Synology to update. Or even a newer docker version. Gitlab may require at least docker 20.10.10 according to their troubleshooting: Install GitLab using Docker | GitLab

The issue is a general Ruby problem, as per the Ruby issue I posted, and someone posted that having a newer kernel resolved their problem. So your only way forward is to ask Synology to fix their system. Or migrate your Gitlab installation away from Synology.

ns1045 · February 23, 2024, 3:47pm

I just opened a ticket on Synology support platform. So I wait curiously for any reaction

dnsmichi · February 26, 2024, 2:31pm

I’m following the topic but do not have specific GitLab insights yet. From my research, the problem is specific to Ruby as the underlaying framework. Updating Ruby to 3.1 is required because 3.0 is end-of-life in March 2024. Work is tracked in Ruby 3.1 (&10034) · Epics · GitLab.org · GitLab

Other projects encountering the urandom error “solved” it by requiring a newer Kernel 4.x+).

Discourse (same Ruby stack as GitLab)
- https://meta.discourse.org/t/kernel-version-upgrade-required-when-manually-updating-discourse/253756/37
- Bootstrap failure during most recent upgrade - #5 by Ed_S - installation - Discourse Meta
Solectrus UI always crashing - `urandom': failed to get urandom (RuntimeError) · Issue #4 · solectrus/hosting · GitHub → Kernel v4 is required · solectrus/hosting@43c6413 · GitHub

It seems that Synology uses a patched 3.x kernel which is EOL by Kernel upstream. According to an employee comment in Reddit - Dive into anything Synology maintains these kernels themselves with patches.

ns1045 · February 26, 2024, 4:05pm

The synology support already replied: it seems not be possible to update the kernel cause of hardware incompatibility issues on my DS916+

Random79 · February 26, 2024, 4:20pm

Do you have a ticket number?
I’d like to inquire if the same is true for my DS216+II .

I wonder how there could possibly be a “hardware incompatibility”
preventing you to compile a newer kernel.
It supports hardware down to the stone age.

dnsmichi · February 27, 2024, 8:09pm

I’ll add screenshots from the Reddit thread which might help dive deeper into the complexity of Synology Kernels and their firmware patches. I’m not defending them, just trying to understand what’s holding them off, or suggest an alternative path. Maybe someone created custom Kernel packages that are not officially supported? I learned something similar with Samsung TVs and SMB support some years ago.

and