Undefined method `gem' for main:Object (NoMethodError)

Hi all,

Our Gitlab installation was running without issues (and we do have a backup), but upon updating recently it failed and started throwing the error on the subject.

This is the update info:

Transaction ID : 9
Begin time     : Fri 14 Aug 2020 01:20:17 PM WEST
Begin rpmdb    : 459:da6d8421630b70c4092183da2b42bcc869066d8b
End time       : Thu 01 Jan 1970 01:00:00 AM CET (-1597407617 seconds)                                                                   
End rpmdb      :
User           : root <root>
Return-Code    : Failure: 1
Releasever     :
Command Line   : update -y
Packages Altered:
 ** Install  python3-unbound-1.7.3-11.el8_2.x86_64            @AppStream                                                                 
 ** Install  unbound-libs-1.7.3-11.el8_2.x86_64               @AppStream                                                                 
 ** Upgrade  nspr-4.25.0-2.el8_2.x86_64                       @AppStream                                                                 
 ** Upgraded nspr-4.21.0-2.el8_0.x86_64                       @@System                                                                   
 ** Upgrade  nss-3.53.1-11.el8_2.x86_64                       @AppStream                                                                 
 ** Upgraded nss-3.44.0-15.el8.x86_64                         @@System                                                                   
 ** Upgrade  nss-softokn-3.53.1-11.el8_2.x86_64               @AppStream                                                                 
 ** Upgraded nss-softokn-3.44.0-15.el8.x86_64                 @@System                                                                   
 ** Upgrade  nss-softokn-freebl-3.53.1-11.el8_2.x86_64        @AppStream                                                                 
 ** Upgraded nss-softokn-freebl-3.44.0-15.el8.x86_64          @@System                                                                   
 ** Upgrade  nss-sysinit-3.53.1-11.el8_2.x86_64               @AppStream                                                                 
 ** Upgraded nss-sysinit-3.44.0-15.el8.x86_64                 @@System                                                                   
 ** Upgrade  nss-tools-3.53.1-11.el8_2.x86_64                 @AppStream                                                                 
 ** Upgraded nss-tools-3.44.0-15.el8.x86_64                   @@System                                                                   
 ** Upgrade  nss-util-3.53.1-11.el8_2.x86_64                  @AppStream                                                                 
 ** Upgraded nss-util-3.44.0-15.el8.x86_64                    @@System                                                                   
 ** Upgrade  NetworkManager-1:1.22.8-5.el8_2.x86_64           @BaseOS                                                                    
 ** Upgraded NetworkManager-1:1.22.8-4.el8.x86_64             @@System                                                                   
 ** Upgrade  NetworkManager-libnm-1:1.22.8-5.el8_2.x86_64     @BaseOS                                                                    
 ** Upgraded NetworkManager-libnm-1:1.22.8-4.el8.x86_64       @@System                                                                   
 ** Upgrade  NetworkManager-team-1:1.22.8-5.el8_2.x86_64      @BaseOS
 ** Upgraded NetworkManager-team-1:1.22.8-4.el8.x86_64        @@System
 ** Upgrade  NetworkManager-tui-1:1.22.8-5.el8_2.x86_64       @BaseOS
 ** Upgraded NetworkManager-tui-1:1.22.8-4.el8.x86_64         @@System
 ** Upgrade  ca-certificates-2020.2.41-80.0.el8_2.noarch      @BaseOS
 ** Upgraded ca-certificates-2019.2.32-80.0.el8_1.noarch      @@System
 ** Upgrade  dbus-1:1.12.8-10.el8_2.x86_64                    @BaseOS
 ** Upgraded dbus-1:1.12.8-9.el8.x86_64                       @@System
 ** Upgrade  dbus-common-1:1.12.8-10.el8_2.noarch             @BaseOS
 ** Upgraded dbus-common-1:1.12.8-9.el8.noarch                @@System
 ** Upgrade  dbus-daemon-1:1.12.8-10.el8_2.x86_64             @BaseOS
 ** Upgraded dbus-daemon-1:1.12.8-9.el8.x86_64                @@System
 ** Upgrade  dbus-libs-1:1.12.8-10.el8_2.x86_64               @BaseOS
 ** Upgraded dbus-libs-1:1.12.8-9.el8.x86_64                  @@System
 ** Upgrade  dbus-tools-1:1.12.8-10.el8_2.x86_64              @BaseOS
 ** Upgraded dbus-tools-1:1.12.8-9.el8.x86_64                 @@System
 ** Upgrade  dnf-4.2.17-7.el8_2.noarch                        @BaseOS
 ** Upgraded dnf-4.2.17-6.el8.noarch                          @@System
 ** Upgrade  dnf-data-4.2.17-7.el8_2.noarch                   @BaseOS
 ** Upgraded dnf-data-4.2.17-6.el8.noarch                     @@System
 ** Upgrade  iptables-1.8.4-10.el8_2.1.x86_64                 @BaseOS
 ** Upgraded iptables-1.8.4-10.el8.x86_64                     @@System
 ** Upgrade  iptables-ebtables-1.8.4-10.el8_2.1.x86_64        @BaseOS
 ** Upgraded iptables-ebtables-1.8.4-10.el8.x86_64            @@System
 ** Upgrade  iptables-libs-1.8.4-10.el8_2.1.x86_64            @BaseOS
 ** Upgraded iptables-libs-1.8.4-10.el8.x86_64                @@System
 ** Upgrade  libdnf-0.39.1-6.el8_2.x86_64                     @BaseOS
 ** Upgraded libdnf-0.39.1-5.el8.x86_64                       @@System
 ** Upgrade  python3-dnf-4.2.17-7.el8_2.noarch                @BaseOS
 ** Upgraded python3-dnf-4.2.17-6.el8.noarch                  @@System
 ** Upgrade  python3-hawkey-0.39.1-6.el8_2.x86_64             @BaseOS
 ** Upgraded python3-hawkey-0.39.1-5.el8.x86_64               @@System
 ** Upgrade  python3-libdnf-0.39.1-6.el8_2.x86_64             @BaseOS
 ** Upgraded python3-libdnf-0.39.1-5.el8.x86_64               @@System
 ** Upgrade  selinux-policy-3.14.3-41.el8_2.5.noarch          @BaseOS
 ** Upgraded selinux-policy-3.14.3-41.el8_2.4.noarch          @@System
 ** Upgrade  selinux-policy-targeted-3.14.3-41.el8_2.5.noarch @BaseOS
 ** Upgraded selinux-policy-targeted-3.14.3-41.el8_2.4.noarch @@System
 ** Upgrade  systemd-239-31.el8_2.2.x86_64                    @BaseOS
 ** Upgraded systemd-239-30.el8_2.x86_64                      @@System
 ** Upgrade  systemd-libs-239-31.el8_2.2.x86_64               @BaseOS
 ** Upgraded systemd-libs-239-30.el8_2.x86_64                 @@System
 ** Upgrade  systemd-pam-239-31.el8_2.2.x86_64                @BaseOS
 ** Upgraded systemd-pam-239-30.el8_2.x86_64                  @@System
 ** Upgrade  systemd-udev-239-31.el8_2.2.x86_64               @BaseOS
 ** Upgraded systemd-udev-239-30.el8_2.x86_64                 @@System
 ** Upgrade  yum-4.2.17-7.el8_2.noarch                        @BaseOS
 ** Upgraded yum-4.2.17-6.el8.noarch                          @@System
 ** Upgrade  gitlab-ee-13.2.4-ee.0.el8.x86_64                 @gitlab_gitlab-ee
 ** Upgraded gitlab-ee-13.2.3-ee.0.el8.x86_64                 @@System

But gitlab-ee-13.2.4-ee.0.el8.x86_64 fails with:

Traceback (most recent call last):
<internal:gem_prelude>:4:in `<internal:gem_prelude>': undefined method `gem' for main:Object (NoMethodError)                           
        1: from <internal:gem_prelude>:3:in `<internal:gem_prelude>'
<internal:gem_prelude>:6:in `rescue in <internal:gem_prelude>': uninitialized constant Gem::LoadError (NameError)                      

Now whenever I do gitlab-ctl always throws that error, and gitlab is not running.

[root@code1 ~]# ruby -v
ruby 2.6.3p62 (2019-04-16 revision 67580) [x86_64-linux]

[root@code1 ~]# cat /etc/os-release 
NAME="CentOS Linux"
VERSION="8 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="8"
PLATFORM_ID="platform:el8"
PRETTY_NAME="CentOS Linux 8 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:8"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-8"
CENTOS_MANTISBT_PROJECT_VERSION="8"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="8"

Using Omnibus Install.

gitlab.rb file is up to date, the diff has been performed against the latest template available on gitlab repo:

[root@code1 gitlab]# diff gitlab.rb gitlab.rb.template 
32c32
< external_url "https://gitlab.domain.io"
---
> external_url 'GENERATED_EXTERNAL_URL'
393,394c393,394
< gitlab_rails['ldap_enabled'] = true
< gitlab_rails['prevent_ldap_sign_in'] = YAML.load_file('/path/to/ldap/config.yml')
---
> # gitlab_rails['ldap_enabled'] = false
> # gitlab_rails['prevent_ldap_sign_in'] = false
2007c2007
< letsencrypt['enable'] = true
---
> # letsencrypt['enable'] = nil