Upgrade from 16.6.2-ce to 16.7.0-ce fails

When I’m trying to update with the omnibus package during the setting up I get this error:

 Setting up gitlab-ce (16.7.0-ce.0) ...
 /opt/gitlab/embedded/lib/ruby/3.1.0/securerandom.rb:75:in `urandom': failed to get urandom (RuntimeError)
         from /opt/gitlab/embedded/lib/ruby/3.1.0/securerandom.rb:75:in `singleton class'
         from /opt/gitlab/embedded/lib/ruby/3.1.0/securerandom.rb:42:in `<module:SecureRandom>'
         from /opt/gitlab/embedded/lib/ruby/3.1.0/securerandom.rb:41:in `<top (required)>'
         from <internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:149:in `require'
         from <internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:149:in `require'
         from /opt/gitlab/embedded/cookbooks/package/libraries/settings_dsl.rb:21:in `<top (required)>'
         from /opt/gitlab/embedded/cookbooks/package/libraries/deprecations.rb:3:in `require_relative'
         from /opt/gitlab/embedded/cookbooks/package/libraries/deprecations.rb:3:in `<top (required)>'
         from <internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:86:in `require'
         from <internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:86:in `require'
         from /opt/gitlab/embedded/service/omnibus-ctl/check_config.rb:18:in `load_file'
         from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/omnibus-ctl-0.6.0/lib/omnibus-ctl.rb:192:in `eval'
         from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/omnibus-ctl-0.6.0/lib/omnibus-ctl.rb:192:in `load_file'
         from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/omnibus-ctl-0.6.0/lib/omnibus-ctl.rb:187:in `block in load_files'
         from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/omnibus-ctl-0.6.0/lib/omnibus-ctl.rb:186:in `each'
         from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/omnibus-ctl-0.6.0/lib/omnibus-ctl.rb:186:in `load_files'
         from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/omnibus-ctl-0.6.0/bin/omnibus-ctl:29:in `<top (required)>'
         from /opt/gitlab/embedded/bin/omnibus-ctl:25:in `load'
         from /opt/gitlab/embedded/bin/omnibus-ctl:25:in `<main>'
 dpkg: error processing package gitlab-ce (--configure):
  installed gitlab-ce package post-installation script subprocess returned error exit status 1
 Errors were encountered while processing:
  gitlab-ce
 E: Sub-process /usr/bin/dpkg returned an error code (1)

It seems that there’s something with urandom. What can I check to see where there’s the problem?

My system is:

sudo cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
NAME="Debian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

Version:

uname -r
3.16.0-4-amd64

Thanks who can help me :slight_smile:

I would say something is wrong with your Debian install. Kernel for Debian 11 right now is 5.10.0-27-amd64. Kernel 3.16 is not a Debian 11 kernel.

1 Like

Thanks, you are right, there’s something strange.
I don’t know why uname gives me the wrong version, but looking at the apt history the last kernel installed is:

Install: linux-image-5.10.0-27-amd64:amd64 (5.10.205-2, automatic)

So I’m a bit lost…

Try:

dpkg -l | grep -i linux-image

then we can see what else is installed kernel-wise. Is this a system that has been upgraded from previous Debian versions? Since kernel 3.16 hints at Debian 8.

1 Like

Yes , that was the first versione then was updated till 11.
Here the result:

 dpkg -l | grep -i linux-image
rc  linux-image-3.16.0-10-amd64       3.16.81-1                      amd64        Linux 3.16 for 64-bit PCs
rc  linux-image-3.16.0-11-amd64       3.16.84-1                      amd64        Linux 3.16 for 64-bit PCs
ii  linux-image-3.16.0-4-amd64        3.16.51-3                      amd64        Linux 3.16 for 64-bit PCs
rc  linux-image-3.16.0-5-amd64        3.16.51-3+deb8u1               amd64        Linux 3.16 for 64-bit PCs
rc  linux-image-3.16.0-6-amd64        3.16.57-2                      amd64        Linux 3.16 for 64-bit PCs
rc  linux-image-3.16.0-7-amd64        3.16.59-1                      amd64        Linux 3.16 for 64-bit PCs
rc  linux-image-3.16.0-8-amd64        3.16.64-2                      amd64        Linux 3.16 for 64-bit PCs
rc  linux-image-3.16.0-9-amd64        3.16.68-2                      amd64        Linux 3.16 for 64-bit PCs
rc  linux-image-4.19.0-20-amd64       4.19.235-1                     amd64        Linux 4.19 for 64-bit PCs (signed)
rc  linux-image-4.9.0-13-amd64        4.9.228-1                      amd64        Linux 4.9 for 64-bit PCs
rc  linux-image-4.9.0-14-amd64        4.9.246-2                      amd64        Linux 4.9 for 64-bit PCs
rc  linux-image-4.9.0-15-amd64        4.9.258-1                      amd64        Linux 4.9 for 64-bit PCs
rc  linux-image-4.9.0-16-amd64        4.9.272-2                      amd64        Linux 4.9 for 64-bit PCs
rc  linux-image-4.9.0-18-amd64        4.9.303-1                      amd64        Linux 4.9 for 64-bit PCs
rc  linux-image-5.10.0-13-amd64       5.10.106-1                     amd64        Linux 5.10 for 64-bit PCs (signed)
rc  linux-image-5.10.0-14-amd64       5.10.113-1                     amd64        Linux 5.10 for 64-bit PCs (signed)
rc  linux-image-5.10.0-15-amd64       5.10.120-1                     amd64        Linux 5.10 for 64-bit PCs (signed)
rc  linux-image-5.10.0-16-amd64       5.10.127-2                     amd64        Linux 5.10 for 64-bit PCs (signed)
rc  linux-image-5.10.0-17-amd64       5.10.136-1                     amd64        Linux 5.10 for 64-bit PCs (signed)
rc  linux-image-5.10.0-18-amd64       5.10.140-1                     amd64        Linux 5.10 for 64-bit PCs (signed)
rc  linux-image-5.10.0-19-amd64       5.10.149-2                     amd64        Linux 5.10 for 64-bit PCs (signed)
rc  linux-image-5.10.0-20-amd64       5.10.158-2                     amd64        Linux 5.10 for 64-bit PCs (signed)
rc  linux-image-5.10.0-21-amd64       5.10.162-1                     amd64        Linux 5.10 for 64-bit PCs (signed)
rc  linux-image-5.10.0-22-amd64       5.10.178-3                     amd64        Linux 5.10 for 64-bit PCs (signed)
rc  linux-image-5.10.0-23-amd64       5.10.179-3                     amd64        Linux 5.10 for 64-bit PCs (signed)
rc  linux-image-5.10.0-24-amd64       5.10.179-5                     amd64        Linux 5.10 for 64-bit PCs (signed)
rc  linux-image-5.10.0-25-amd64       5.10.191-1                     amd64        Linux 5.10 for 64-bit PCs (signed)
ii  linux-image-5.10.0-26-amd64       5.10.197-1                     amd64        Linux 5.10 for 64-bit PCs (signed)
ii  linux-image-5.10.0-27-amd64       5.10.205-2                     amd64        Linux 5.10 for 64-bit PCs (signed)
ii  linux-image-amd64                 5.10.205-2                     amd64        Linux for 64-bit PCs (meta-package)

You want to be doing a cleanup. First I would reboot and from the grub menu boot from the latest 5.10 kernel. Then once that is done, you need to clean up removed packages by doing:

apt-get autoremove --purge

After that, you want to look at cleaning up any leftover Debian 8,9,10 packages by doing:

dpkg -l | grep -i deb8

repeat same command changing 8 to 9 and 10. Any of the packages listed from those commands need to be removed. A quick and easy way is:

apt-get remove --purge `dpkg -l | grep -i deb8 | awk '{print $2}'`

again, change deb8 to deb9 and deb10 later when you repeat the commands. You do however need to be booted into a Debian 11 5.10.x kernel before you do that though.

The potential for lack of urandom, is because you are not using the kernel that is expected to be used, and so it’s not showing up. Although from what I remember I’m pretty sure urandom existed even in 2.x and 3.x kernels but I cannot verify that right now.

Either way, you need your system to be sane and clean from old packages that shouldn’t be there as they could be causing your problems.

2 Likes

Thanks a lot for your help.
Is a virtual machine so I need to ask the IT guys on how to procede.

When done I’ll update and see what happens.

Also, make sure that the gitlab-ce package that you have installed is actually for Debian 11 as well. As if the repo was for Debian 10 it will have dependencies from Debian 10. That means using some of the commands I gave above could end up removing Gitlab.

Basically as long as whoever is doing this pays attention to the commands and what packages are being removed, then it shouldn’t be a problem. If the gitlab package shows up in the list of packages to be removed, then you need to stop and not run that command. You should have under /etc/apt/sources.list.d/ a list file for gitlab-ce so check the contents of this to make sure it’s using bullseye which is Debian 11.

2 Likes

Thanks, checked and the package for gitlab is referred to debian 11 (bullseye).
So for that I’m ok.

Need to boot the VM with the correct kernel.

Thanks again!

1 Like

So, was a bit difficult, wasn’t installed grub but syslinux.
I could boot with the new kernel, but the gitlab 16.6.2 couldn’t start, the puma thread and another thread occupied 100% of the cpu.
Tried anyway to update to the 16.7.0 and 16.7.2 but got some error trying to do the sql backup.

Preparing to unpack .../gitlab-ce_16.7.0-ce.0_amd64.deb ...
gitlab preinstall: Automatically backing up only the GitLab SQL database (excluding everything else!)
rake aborted!
NoMethodError: undefined method `include?' for 0:Integer
/opt/gitlab/embedded/service/gitlab-rails/config/settings.rb:56:in `build_gitlab_shell_ssh_path_prefix'
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/1_settings.rb:923:in `<top (required)>'
/opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:7:in `<top (required)>'
<internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.0.0/rubygems/core_ext/kernel_require.rb>:38:in `require'
<internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.0.0/rubygems/core_ext/kernel_require.rb>:38:in `require'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/helpers.rake:7:in `block in <top (required)>'
/opt/gitlab/embedded/bin/bundle:25:in `load'
/opt/gitlab/embedded/bin/bundle:25:in `<main>'
Tasks: TOP => environment
(See full trace by running task with --trace)
gitlab preinstall:
gitlab preinstall: Database backup failed! If you want to skip this backup, run the following command and try again:
gitlab preinstall:
gitlab preinstall:  sudo touch /etc/gitlab/skip-auto-backup
gitlab preinstall:
dpkg: error processing archive /var/cache/apt/archives/gitlab-ce_16.7.0-ce.0_amd64.deb (--unpack):
 new gitlab-ce package pre-installation script subprocess returned error exit status 1
Errors were encountered while processing:
 /var/cache/apt/archives/gitlab-ce_16.7.0-ce.0_amd64.deb
E: Sub-process /usr/bin/dpkg returned an error code (1)

Got back to the old kernel with 16.6.2 and is working, for now I’ll leave as it.

Next week I’ll do a new VM with an updated linux image and see how it goes, probably during some system upgrade something went wrong. Better to start from 0.

Thanks for your help!

Normal Debian installs should be Grub. They at least have been since I started using Debian from 4.x. So not sure why your install is configured in a strange way booting with syslinux (not including the old kernel or old distro packages still installed).

Might be best to install Gitlab 16.6.2 on a clean new Debian 11 install and then use the Gitlab Backup/Restore documentation to move away from your old problematic server.

1 Like

I’ll do so, thanks.
Do you suggest debian or ubuntu?

I use and prefer Debian on all of mine :slight_smile:

But, assuming you install the same Gitlab version as your old server, so 16.6.2, you can restore it, even if it’s Ubuntu, Rocky Linux, or any of the other supported distros with a gitlab-ce package.

The best distro is the one you are happy using on a day-to-day basis, and don’t have to learn too much when switching.

1 Like

Same error on “Ubuntu 22.04.3 LTS”, is that a bug? I"m on a proprietary kernel 3.14 based, is urandom not there?

I have the same issue. My Gitlab-ce is running on a Synology with a 3.10 kernel. A kernel update is not possible.

If you are running Ubuntu 22.04 with a 3.x kernel, then your system has either been upgraded incorrectly - or especially since you mention running a proprietary kernel makes me ask why are you running a proprietary kernel and for what reason? That means not supplied by Ubuntu obviously since their kernel’s are not proprietary and rather open-source. It’s not a bug, it means you are not running your system properly after it’s been upgraded wrong.

Docker doesn’t run a kernel. Docker uses the host kernel so your problem is irrelevant here and rather something else related with your Synology install.

Hi @iwalker,
Thanks for the answer, that’s what I’m thinking. The kernel of the server is given as is, and it runs for now. Gitlab never complains over updates, that’s the first time, but perhaps it reaches the max the kernel can give. I’ve asked the tech of the server why there is no recent kernel available. I’m waiting for an answer. Meanwhile, I’ve reverted to 16.6.
Ramel

1 Like

Ok, just to inform that with a new installation (ubuntu 5.15.0-91-generic) and migrating the data from the old server, the upgrade from 16.6.2 to 16.7.2 went without a problem (also the upgrade to 16.8.0).

Thanks for the support!

1 Like