GitLab upgrade 14.0.12 to any version above (14.1.x) fails with broken assets and 500 errors in GUI

Hello All,

I am starting at 14.0.12. It has been upgraded from 14.0.3 without a problem. Now if I upgrade to any version of 14.1x. the assets are broken.

I can see the errors in the /var/log/gitlab/gitlab-rails/production.log as follows …

Started GET “/users/sign_in” for 10.2.14.170 at 2022-09-22 17:13:43 +0100
Processing by SessionsController#new as HTML
Rendered layout layouts/devise.html.haml (Duration: 18.2ms | Allocations: 8564)
Completed 500 Internal Server Error in 46ms (ActiveRecord: 1.6ms | Elasticsearch: 0.0ms | Allocations: 13459)

ActionView::Template::Error (No such file or directory @ rb_sysopen - /opt/gitlab/embedded/service/gitlab-rails/public/assets/themes/theme_indigo-356df9446472d3f379aa66e5b212f2a89799ba6b0fd209831ee4b01089b94f7e.css):
2: - startup_filename = local_assigns.fetch(:startup_filename, nil) || startup_filename_default
3:
4: %style
5: = Rails.application.assets_manifest.find_sources(“themes/#{user_application_theme_css_filename}.css”).first.to_s.html_safe if user_application_theme_css_filename
6: = Rails.application.assets_manifest.find_sources(“startup/startup-#{startup_filename}.css”).first.to_s.html_safe app/views/layouts/_startup_css.haml:5:in `each’

When I look in the directory /opt/gitlab/embedded/service/gitlab-rails/public/assets/themes

the file is theme_indigo-356df9446472d3f379aa66e5b212f2a89799ba6b0fd209831ee4b01089b94f7e.css is not there but a new version of theme_indigo is present.

It would appear that GitLab does not know about the new version of theme_indigo etc. I am using GitLab from rpm on rhel7 so the assets should be compiled in.

If I manually copy in old versions I can get the login screen but obviously this is not a good fix as there are loads of assets missing.

Any ideas on how to fix this or what could be wrong?

Best Regards,

Kevin.

Hello All,

I have got to the bottom of this. I am sharing the solution here just in case this happens to anyone else.

At the bottom of the upgrade to 14.1.8 I see the following message.

WARNING:
GitLab discovered stale file(s) from a previous install that need to be cleaned up.
The following files need to be removed:

/opt/gitlab/embedded/service/gitlab-rails/public/assets/.sprockets-manifest-d2a57e5758698a61e3a09bfd95948496.json

If I remove that file and restart gitlab then the upgrade from 14.0.12 to 14.1.8 works. I can then upgrade to latest following the correct steps got from here → Upgrading GitLab | GitLab

Best Regards,

Kevin.

1 Like