Startup loop after 14.1.4 -> 14.2.0 upgrade (debian, omnibus)

Yes, yet another one of these posts.

I am using gitlab-ce on debian buster using the gitlab repository/omnibus. While using apt my installation was upgraded from 14.0.x to the latest version (14.2.3). As a result I only got HTTP/502 on the WebUI. I rolled back and manuall< installed all incremental upgrades which narrowed the problem down to the upgrade 14.1.4 → 14.2.0.

After the Upgrade I’ve got the following envigonment:

 Ruby:         ruby 2.7.2p137 (2020-10-01 revision 5445e04352) [x86_64-linux]
 GitLab:       14.2.0 (d678b7c987f) FOSS
 GitLab Shell: 13.19.1
 PostgreSQL:   13.3

(Postgres 12.x was manually updated in this test, 12.x showed the same error)

gitlab-ctl reconfigure, gitlab-ctl upgrade and gitlab-rake db:migrate all finish without errors, gitlab-ctl status also doesn’t show any errors. gitlab-rake db:migrate:status confirms all migrations to be “up”. As said WebUI is responding but stuck at 502. I see puma and sidekiq both running at 100% CPU and changing PIDs, so probably crashing and respawning. I tail -f’d all *log-files in /var/log/gitlab and did only get some startup messages, but no errors.

Looking for background tasks using gitlab-rails runner -e production 'puts Gitlab::BackgroundMigration.remaining' (or any other gitlab-rails-command like gitlab-rails console) yields:
[...]file /opt/gitlab/embedded/service/gitlab-rails/lib/migrate.rb to define constant Migrate, but didn't (Zeitwerk::NameError).

I’m not quite sure where to go from here. As far as I can tell all migrations should be completed, so why exactly is rake crashing at the migration stage? Any ideas or tips where I can look next?`

Thanks
Florian

2 Likes

We resolved a similar issue be going 14.1.5 → 14.2.0, is there a reason you went 14.1.4 straight to 14.2 when you were trying incremental upgrades?

As I’m having the similaire issues trying to upgrade 14.1.* → 14.2.* (I’ve tried all versions), I can confirm that 14.1.5 → 14.2.3 still leads to this last migration error, preventing Gitlab from being usable.
So I’m currently stucked at 14.1.5 as I haven’t been able to find a workaround anywhere.
I did also faced other migration issues that indeed are now resolved in the latest 14.2 release.

==> /var/log/gitlab/puma/current <==
2021-09-13_14:57:55.66244 {"timestamp":"2021-09-13T14:57:55.662Z","pid":487464,"message":"! Unable to load application: Zeitwerk::NameError: expected file /opt/gitlab/embedded/service/gitlab-rails/lib/migration.rb to define constant Migration, but didn't"}
2021-09-13_14:57:55.66260 bundler: failed to load command: puma (/opt/gitlab/embedded/bin/puma)
2021-09-13_14:57:55.66268 Zeitwerk::NameError: expected file /opt/gitlab/embedded/service/gitlab-rails/lib/migration.rb to define constant Migration, but didn't
2021-09-13_14:57:55.66269   /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/zeitwerk-2.4.2/lib/zeitwerk/loader/callbacks.rb:18:in `on_file_autoloaded'
2021-09-13_14:57:55.66269   /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/zeitwerk-2.4.2/lib/zeitwerk/kernel.rb:27:in `block in require'

Hm, didn’t see 14.1.5 in the Debian repo back then

I could solve the problem by completely purging the package and reinstalling Gitlab 14.1.4. Since the data itself wasn’t touched I didn’t loose any repositories or configuration. After this the upgrade to 14.2 worked as expected.

apt remove gitlab-ce
apt purge gitlab-ce
rm -Rv /opt/gitlab/
apt install gitlab-ce=14.1.4-ce.0
gitlab-ctl reconfigure
apt full-upgrade
gitlab-ctl reconfigure
1 Like