Docker container with gitlab version >=16.7.0 can not be started due ruby error with urandom on Synology

When I want to run any gitlab-ce docker image with versions > 16.7.0 I get an error that container could not be started. Ruby can not use securerandom due file access problems of dev/urandom. similar to https://bugs.ruby-lang.org/issues/14716.
https://forum.gitlab.com/t/upgrade-from-16-6-2-ce-to-16-7-0-ce-fails/98104

/opt/gitlab/embedded/lib/ruby/3.1.0/securerandom.rb:75:in `urandom': failed to get urandom (RuntimeError)
	from /opt/gitlab/embedded/lib/ruby/3.1.0/securerandom.rb:75:in `singleton class'
	from /opt/gitlab/embedded/lib/ruby/3.1.0/securerandom.rb:42:in `<module:SecureRandom>'
	from /opt/gitlab/embedded/lib/ruby/3.1.0/securerandom.rb:41:in `<top (required)>'
	from <internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:149:in `require'
	from <internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:149:in `require'
	from /opt/gitlab/embedded/cookbooks/package/libraries/settings_dsl.rb:21:in `<top (required)>'
	from /opt/gitlab/embedded/cookbooks/package/libraries/deprecations.rb:3:in `require_relative'
	from /opt/gitlab/embedded/cookbooks/package/libraries/deprecations.rb:3:in `<top (required)>'
	from <internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:86:in `require'
	from <internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:86:in `require'
	from /opt/gitlab/embedded/service/omnibus-ctl/check_config.rb:18:in `load_file'
	from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/omnibus-ctl-0.6.0/lib/omnibus-ctl.rb:192:in `eval'
	from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/omnibus-ctl-0.6.0/lib/omnibus-ctl.rb:192:in `load_file'
	from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/omnibus-ctl-0.6.0/lib/omnibus-ctl.rb:187:in `block in load_files'
	from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/omnibus-ctl-0.6.0/lib/omnibus-ctl.rb:186:in `each'
	from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/omnibus-ctl-0.6.0/lib/omnibus-ctl.rb:186:in `load_files'
	from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/omnibus-ctl-0.6.0/bin/omnibus-ctl:29:in `<top (required)>'
	from /opt/gitlab/embedded/bin/omnibus-ctl:25:in `load'
	from /opt/gitlab/embedded/bin/omnibus-ctl:25:in `<main>'

With 16.6.4 and 16.6.5 the container starts up.
Docker version: 20.10.3.1308 (due preinstalled on synology NAS)
System: 3.10.108 #42962 SMP Mon May 29 14:35:41 CST 2023 x86_64 GNU/Linux synology_braswell_916+
container imager: gitlab/gitlab-ce:16.7.0-ce.0 and gitlab/gitlab-ce:16.8.0-ce.0

2 Likes

I have the same issue on my Synology NAS (DS916+) all patches/updates installed:
Docker version 20.10.3, build 55f0773
Linux version 3.10.108 #42962 SMP Mon May 29 14:35:41 CST 2023

With [16.6.5-ee.0] I was able to start GitLab without this issue.

Same here and still using DSM 7.1s docker. I guess the world is really pushing into that 7.2 update because with 7.2s container everything runs smoothly

1 Like

Unfortunately an update to 7.2 did not resolved the issue. The kernel remains on 3.10.*
So we must hope, that synology updates the kernel version soon

1 Like

Same issue here.
One 2021 Synology model updated to DSM DSM 7.2.1-69057 Update 4 ( did the update fine
but two older 2016 models that had to be manually updatded to DSM 7.2.1-69057 Update 4 show the same error as above with 16.7.0 and “latest” gitlab-ce Docker images.
16.6.6 is the newest that works.

2 Likes

This has become critical.
There is a security release of 16.9.1/16.8.3/16.7.6 that doesn’t have a 16.6 release like the 2 security issues before.
So now we have to update but the update doesn’t run because the Kernel version stays at 3.10.108 .

1 Like

Have you opened a support issue with Synology? Since Gitlab works in Docker pretty much on every Linux distro, and yet the problems here are only related to running it on Synology - it would seem that Synology is the problem here in something that they have excluded for some bizarre reason. I expect pretty much any other container app that works in a similar way like Gitlab will also fail on Synology due to something that has been omitted by Synology.

Also this issue: /usr/local/lib/ruby/3.1.0/securerandom.rb:75:in `urandom': failed to get urandom (RuntimeError) · Issue #383 · docker-library/ruby · GitHub suggests requiring a newer kernel as already mentioned - which you will have to ask Synology to update. Or even a newer docker version. Gitlab may require at least docker 20.10.10 according to their troubleshooting: Install GitLab using Docker | GitLab

The issue is a general Ruby problem, as per the Ruby issue I posted, and someone posted that having a newer kernel resolved their problem. So your only way forward is to ask Synology to fix their system. Or migrate your Gitlab installation away from Synology.

I just opened a ticket on Synology support platform. So I wait curiously for any reaction :slight_smile:

2 Likes

I’m following the topic but do not have specific GitLab insights yet. From my research, the problem is specific to Ruby as the underlaying framework. Updating Ruby to 3.1 is required because 3.0 is end-of-life in March 2024. Work is tracked in Ruby 3.1 (&10034) · Epics · GitLab.org · GitLab

Other projects encountering the urandom error “solved” it by requiring a newer Kernel 4.x+).

  1. Discourse (same Ruby stack as GitLab)
  2. Solectrus UI always crashing - `urandom': failed to get urandom (RuntimeError) · Issue #4 · solectrus/hosting · GitHubKernel v4 is required · solectrus/hosting@43c6413 · GitHub

It seems that Synology uses a patched 3.x kernel which is EOL by Kernel upstream. According to an employee comment in Reddit - Dive into anything Synology maintains these kernels themselves with patches.

1 Like

The synology support already replied: it seems not be possible to update the kernel cause of hardware incompatibility issues on my DS916+

Do you have a ticket number?
I’d like to inquire if the same is true for my DS216+II .

I wonder how there could possibly be a “hardware incompatibility”
preventing you to compile a newer kernel.
It supports hardware down to the stone age.

I’ll add screenshots from the Reddit thread which might help dive deeper into the complexity of Synology Kernels and their firmware patches. I’m not defending them, just trying to understand what’s holding them off, or suggest an alternative path. Maybe someone created custom Kernel packages that are not officially supported? I learned something similar with Samsung TVs and SMB support some years ago.

and

@dnsmichi the problem is with ruby 3.1.x, an update to ruby 3.2.x would fix it as you can see in Upgrade Ruby to 3.2.x to remove bug introduced on Ruby 3.1.x · Issue #2886 · sameersbn/docker-gitlab · GitHub

So, is it possible to build intermediate Docker images for the requried update steps of GitLab for Synology using Ruby 3.2.x?
Or are there breaking changes between Ruby 3.1 and 3.2 tha prevent this?

Does a current GitLab use Ruby 3.2, so we could export everything and import it into a new GitLab? (not being able to perform the intermediate update steps)

Depends which Docker image is used. sameersbn/docker-gitlab builds GitLab from source, following the Dockerfile executing docker-gitlab/assets/build/install.sh at master · sameersbn/docker-gitlab · GitHub The source install also applies some patches which modify the upstream sources, docker-gitlab/assets/build/patches at master · sameersbn/docker-gitlab · GitHub for example removing the seed functionality that is broken with Ruby 3.1.x.

The GitLab provided container images uses the Omnibus package, which has been updated to Ruby 3.1.5 and prepares support for 3.2.5 in Upgrade to Ruby 3.1.5 and add support for Ruby 3.2.5 (!7591) · Merge requests · GitLab.org / omnibus-gitlab · GitLab The Docker setup scripts do not imply that the default Ruby version 3.1.5 was changed recently. I don’t know the plans here but would encourage you to opening an issue to upgrade Ruby 3.2.x in Docker/Omnibus if needed.

Is there a point asking Docker for that?
With the speed new versions are coming up,
16.7 is outside of their window of maintained versions already.

Docker is only the tool to package software in a container image. You would need to ask the maintainers of the specific docker images to bump the versions. I’ve seen that Release 17.0.1 · sameersbn/docker-gitlab · GitHub was released last week for example.

My mistake.
I meant asking GitLab

You can use this pre-selected issue template.

I aborted after it wanted
not only my github account (to create a different account from this forum one),
verify the email adress I used there (wich is NOT the one I would want to use with GitLab)
and then also wanted to have a phone-number.
That’s WAY too much personal information. I will not give it my landline (since I only use it for T.38 fax) and certainly not my cellphone number (that gives away what messenger services I have accounts with).