Strange "Internal Server Error" from API after upgrade

How to Use GitLab System Administration
1

We have two (load-balanced) app servers serving GitLab for us.
Earlier today I upgraded then from 15.10 to 15.11. That was mostly succesful, but afterwards one of the servers return an “Internal Server Error” every time I query /api/v4/application/settings, other API endpoints work fine, and it work on the other server.
What can cause that?

I debugged a little further, and found that something is written into /var/log/gitlab/gitlab-rails/api_json.log for the requests. Making similar requests on both servers, compared those and removing all the stuff that seems irrelevant (stuff that relates to timing, memory/database usage, …), the difference seems to be that the following lines are logged on the server where the request doesn’t work:

  "exception.class": "NoMethodError",
  "exception.message": "API::Entities::ApplicationSetting missing attribute `silent_mode_enabled' on #<ApplicationSetting:0x00007feebd0392e8>",
  "exception.backtrace": [
    "lib/gitlab/json.rb:130:in `dump'",
    "lib/gitlab/json.rb:130:in `adapter_dump'",
    "lib/gitlab/json.rb:52:in `dump'",
    "lib/gitlab/json.rb:194:in `call'",
    "lib/api/api_guard.rb:222:in `call'",
    "lib/gitlab/middleware/memory_report.rb:13:in `call'",
    "lib/gitlab/middleware/speedscope.rb:13:in `call'",
    "lib/gitlab/database/load_balancing/rack_middleware.rb:23:in `call'",
    "lib/gitlab/jira/middleware.rb:19:in `call'",
    "lib/gitlab/middleware/go.rb:20:in `call'",
    "lib/gitlab/etag_caching/middleware.rb:21:in `call'",
    "lib/gitlab/middleware/query_analyzer.rb:11:in `block in call'",
    "lib/gitlab/database/query_analyzer.rb:37:in `within'",
    "lib/gitlab/middleware/query_analyzer.rb:11:in `call'",
    "lib/gitlab/middleware/multipart.rb:173:in `call'",
    "lib/gitlab/middleware/read_only/controller.rb:50:in `call'",
    "lib/gitlab/middleware/read_only.rb:18:in `call'",
    "lib/gitlab/middleware/same_site_cookies.rb:27:in `call'",
    "lib/gitlab/middleware/basic_health_check.rb:25:in `call'",
    "lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'",
    "lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'",
    "lib/gitlab/middleware/request_context.rb:21:in `call'",
    "lib/gitlab/middleware/webhook_recursion_detection.rb:15:in `call'",
    "config/initializers/fix_local_cache_middleware.rb:11:in `call'",
    "lib/gitlab/middleware/compressed_json.rb:37:in `call'",
    "lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call'",
    "lib/gitlab/middleware/sidekiq_web_static.rb:20:in `call'",
    "lib/gitlab/metrics/requests_rack_middleware.rb:79:in `call'",
    "lib/gitlab/middleware/release_env.rb:13:in `call'"
  ],

Does that help anybody?
2

Restarting (not just sending it a SIGHUP) puma, fixed this.

1 Like