500 Error after upgrade to Gitlab 14.7 from 14.2.2 gitlab-ce omnibus install

General
1

I ran an sudo apt-get update followed by sudo apt-get upgrade. This upgraded my Gitlab to 14.7 from 14.2.2 and now, I can log in to our gitlab server, but, after logging in all I can see is a 500 error.
Here is my system information:

System information
System: Ubuntu 18.04
Current User: -------------- (i removed this for posting)
Using RVM: no
Ruby Version: 2.7.5p203
Gem Version: 3.1.4
Bundler Version:2.1.4
Rake Version: 13.0.6
Redis Version: 6.0.16
Git Version: 2.33.1.
Sidekiq Version:6.3.1
Go Version: unknown

GitLab information
Version: 14.7.0
Revision: abbf44bd6cf
Directory: /opt/gitlab/embedded/service/gitlab-rails
DB Adapter: PostgreSQL
DB Version: 12.7
URL: -------------- (i removed this for posting)
HTTP Clone URL: -------------- (i removed this for posting)
SSH Clone URL: -------------- (i removed this for posting)
Using LDAP: yes
Using Omniauth: yes
Omniauth Providers:

Below is the traceback error that I got from the log files by following the correlation id on the web page with the error. You should be aware that I used gitlab administration function to customize my home page icon. Any suggestions for how I can fix this?

{“method”:“GET”,“path”:"/",“format”:“html”,“controller”:“RootController”,“action”:“index”,“status”:500,“time”:“2022-01-31T18:03:37.455Z”,“params”:,“correlation_id”:“01FTRMWY6QDP8T346X7NZCWRKD”,“meta.user”:“username”,“meta.caller_id”:“RootController#index”,“meta.remote_ip”:“10.21.34.7”,“meta.feature_category”:“projects”,“meta.client_id”:“user/2”,“remote_ip”:“10.21.34.7”,“user_id”:2,“username”:“username”,“ua”:“Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:96.0) Gecko/20100101 Firefox/96.0”,“queue_duration_s”:0.01521,“request_urgency”:“default”,“target_duration_s”:1,“redis_calls”:4,“redis_duration_s”:0.000771,“redis_read_bytes”:483,“redis_write_bytes”:1332,“redis_cache_calls”:2,“redis_cache_duration_s”:0.000417,“redis_cache_read_bytes”:404,“redis_cache_write_bytes”:125,“redis_shared_state_calls”:1,“redis_shared_state_duration_s”:0.000126,“redis_shared_state_write_bytes”:53,“redis_sessions_calls”:1,“redis_sessions_duration_s”:0.000228,“redis_sessions_read_bytes”:79,“redis_sessions_write_bytes”:1154,“db_count”:15,“db_write_count”:0,“db_cached_count”:1,“db_replica_count”:0,

"db_primary_count":15,"db_replica_cached_count":0,"db_primary_cached_count":1,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_replica_wal_cached_count":0,

"db_primary_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.087,"cpu_s":0.322123,"mem_objects":53848,"mem_bytes":5131432,

"mem_mallocs":12865,"mem_total_bytes":7285352,"pid":6021,

"exception.class":"ActionView::Template::Error",

"exception.message":"undefined method `show_customize_homepage_banner?' for #<ActionView::Base:0x00000000095cb8>\nDid you mean?  show_invite_banner?",

"exception.backtrace":["app/views/root/index.html.haml:1","app/controllers/application_controller.rb:133:in `render'",

    "app/controllers/dashboard/projects_controller.rb:63:in `block in render_projects'","lib/gitlab/gitaly_client.rb:306:in `allow_n_plus_1_calls'",

    "app/controllers/dashboard/projects_controller.rb:62:in `render_projects'","app/controllers/dashboard/projects_controller.rb:22:in `block (2 levels) in index'",

    "app/controllers/dashboard/projects_controller.rb:20:in `index'","app/controllers/root_controller.rb:25:in `block in index'",

    "lib/gitlab/gitaly_client.rb:306:in `allow_n_plus_1_calls'","app/controllers/root_controller.rb:23:in `index'",

    "app/controllers/application_controller.rb:490:in `set_current_admin'","lib/gitlab/session.rb:11:in `with_session'",

    "app/controllers/application_controller.rb:481:in `set_session_storage'","lib/gitlab/i18n.rb:105:in `with_locale'",

    "lib/gitlab/i18n.rb:111:in `with_user_locale'","app/controllers/application_controller.rb:475:in `set_locale'",

    "app/controllers/application_controller.rb:469:in `set_current_context'","lib/gitlab/metrics/elasticsearch_rack_middleware.rb:16:in `call'",

    "lib/gitlab/middleware/rails_queue_duration.rb:33:in `call'","lib/gitlab/middleware/speedscope.rb:13:in `call'",

    "lib/gitlab/request_profiler/middleware.rb:17:in `call'","lib/gitlab/database/load_balancing/rack_middleware.rb:23:in `call'",

    "lib/gitlab/metrics/rack_middleware.rb:16:in `block in call'","lib/gitlab/metrics/web_transaction.rb:46:in `run'",

    "lib/gitlab/metrics/rack_middleware.rb:16:in `call'","lib/gitlab/jira/middleware.rb:19:in `call'","lib/gitlab/middleware/go.rb:20:in `call'",

    "lib/gitlab/etag_caching/middleware.rb:21:in `call'","lib/gitlab/middleware/multipart.rb:173:in `call'",

    "lib/gitlab/middleware/read_only/controller.rb:50:in `call'","lib/gitlab/middleware/read_only.rb:18:in `call'",

    "lib/gitlab/middleware/same_site_cookies.rb:27:in `call'","lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'",

    "lib/gitlab/middleware/basic_health_check.rb:25:in `call'","lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'",

    "lib/gitlab/middleware/request_context.rb:21:in `call'","lib/gitlab/middleware/webhook_recursion_detection.rb:15:in `call'",

    "config/initializers/fix_local_cache_middleware.rb:11:in `call'","lib/gitlab/middleware/compressed_json.rb:26:in `call'",

    "lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call'","lib/gitlab/middleware/sidekiq_web_static.rb:20:in `call'",

    "lib/gitlab/metrics/requests_rack_middleware.rb:75:in `call'","lib/gitlab/middleware/release_env.rb:13:in `call'"],

    "db_duration_s":0.08523,"view_duration_s":0.0,"duration_s":0.38535}
2

It’s only the web interface that is broken currently, I can explore the repository with python and the gitlab api, it breaks because it does not know the method “show_customize_homepage_banner” as shown above in my original post…

3

I figured out what I did wrong. If you do a sudo apt-get upgrade, then dont read carefully, the latest version of gitlab will install overtop your current version. :frowning: this is what happened to me.

4

Yes, this is a little unfortunate. I almost did this once too and would have trashed my installation as it would have gone from 12.9.3 to 13.x. Ideally a mechanism should be put into the pre-install scripts so that it doesn’t install 14.7 unless you meet a specific minimum version number. That would mean a bit more work for the Gitlab team when preparing packages, but would help protect installations from breaking. Since then, the only real solution is restoring from backup to get your install working at 14.2.2 again, and then following the upgrade path.

1 Like
5

Thanks for the reply, yes, you are correct, that is what I had to do. Fortunately I had backups of /var/opt/gitlab, /etc/gitlab/gitlab-secrets.json. I tried reinstallaing 14.7.0, running all sorts of commands to check the database. I was able to get access through my api just the web interface would not work.

So, I downgraded to 14.2.0 then back up to 14.2.2, running checks etc between, copying my backup of /var/opt/gitlab back into place at this point, then finally upgraded again to 14.7.0 and finally, after many 500 errors, googling and applying many gitlab-rake commands, much prayer, re-installing, copying my backups into place, restarting, upgrading etc my system came back to life. I still am not sure exactly how, nor could I reproduce my steps. In the process I learned much more about the system and my ability to make errors. I am also impressed that the system was able to tolerate all my mistakes and eventually come back to life! I am grateful to be back.

3 Likes
6

If this upgrade was unintended and ended up happening as a consequence of your upgrading your server’s OS and other packages, think about using sudo apt-mark hold gitlab-ee which will keep GitLab at its installed version during other upgrade processes.

If you are using GitLab CE, change the package name I used above to the right one.

1 Like