Getting this on a daily basis. This seems to be a common problem with no solution that I’ve seen.
Have to login via SSH and restart, every time.
1 Like
Hi @skunkworks_ca Please add some more information about the 500 error from the logs. I’m going to assume that you have installed GitLab via the omnibus package.
When you next experience the 500 error, tail the logs on the server to gather the stack trace so that we can determine the source of the error:
sudo gitlab-ctl tail
This will allow the log entries to be displayed in the terminal when you reproduce the error. Please copy the errors that you discover here, ensuring to remove any sensitive data before posting 
Thanks!
Hi @markglenfletcher, Making sure I understood your instructions properly…
When next we get the error, log in to the server as root via SSH and run that command which will output info needed to diagnose further?
Run the command and reproduce the 500 error to log it to the terminal. It’ll save searching through the log files for the errors. From there it’ll be much easier to help. Thanks!
If the error is not easy to reproduce on command then you may need to search the logs at:
/var/log/gitlab/gitlab-rails/production.log
Search for ‘Internal Server Error’
If the error is not easy to reproduce on command then you may need to search the logs at:
/var/log/gitlab/gitlab-rails/production.log
Search for ‘Internal Server Error’
String ‘Internal Server Error’ not present in production.log (Checked immediately after the error.)
Interestingly… this time the 500 error went away on a browser refresh a minute later.
Please update if you find the error in the logs, tail the last 500 lines of the production.log straight after you see the 500 error. You should be able to find it 
This is happening now on the main gitlab site, its quite frequent actually.
Hi Gtilab,
We are testing Gitlab to export a single file from a repository at Gitlab.com. We use Python to do this.
It works great, but we do get regular Error 500’s. Logging in at gitlab.com won’t work either then.
The error messages we receive let us know that the problem is at Gitlab’s end. Is there anythin on our end we can do to prevent these messages?
We have the same issue when user click save. Here’s log
Started PATCH "/yi.liu/testwebhook/hooks/1" for 10.71.2.83 at 2019-09-11 04:08:15 +0000
Processing by Projects::HooksController#update as HTML
Parameters: {"utf8"=>"✓", "authenticity_token"=>"[FILTERED]", "hook"=>"[FILTERED]", "namespace_id"=>"yi.liu", "project_id"=>"testwebhook", "id"=>"1"}
Completed 500 Internal Server Error in 70ms (ActiveRecord: 6.7ms)
ActionView::Template::Error (undefined method `any?' for nil:NilClass):
4: Recent Deliveries
5: %p When an event in GitLab triggers a webhook, you can use the request details to figure out if something went wrong.
6: .col-lg-9
7: - if hook_logs.any?
8: %table.table
9: %thead
10: %tr
app/views/projects/hook_logs/_index.html.haml:7:in `_app_views_projects_hook_logs__index_html_haml__2843519955585731005_69817950232900'
app/views/projects/hooks/edit.html.haml:20:in `_app_views_projects_hooks_edit_html_haml___1578691917775146108_69817949225660'
app/controllers/application_controller.rb:117:in `render'
app/controllers/projects/hooks_controller.rb:38:in `update'
lib/gitlab/session.rb:11:in `with_session'
Our version is 12.0.8-ce
Now we have found we have a work around for this:
root@10:/# gitlab-rails console
-------------------------------------------------------------------------------------
GitLab: 12.0.8 (36325e121f4)
GitLab Shell: 9.3.0
PostgreSQL: 10.7
-------------------------------------------------------------------------------------
Loading production environment (Rails 5.1.7)
irb(main):001:0> ApplicationSetting.current.reset_runners_registration_token!
=> true
3 Likes
Installed the 13.1.2 (2e7a6f64b07) version of Gitlab-CE and keep getting the 500 errors when accessing the Admin / Settings / General tab. Reconfigure / restart / reboot did not help. Have not noticed any other problems at this time. Only the General Tab is constantly giving a 500 error.
Just switched from OKTA SAML to AzureAD - not sure if this is adding to the problem. Disabled Okta and AzureAd but this did not help.
Here is part of the log - /var/log/gitlab/gitlab-rails/production.log
Processing by Admin::ApplicationSettingsController#general as HTML
Completed 500 Internal Server Error in 22ms (ActiveRecord: 2.4ms | Elasticsearch: 0.0ms | Allocations: 10194)
ActionView::Template::Error (undefined method `first’ for nil:NilClass):
12: = render_if_exists ‘admin/application_settings/default_project_deletion_adjourned_period_setting’, form: f
13: .form-group.visibility-level-setting
14: = f.label :default_project_visibility, class: ‘label-bold’
15: = render(‘shared/visibility_radios’, model_method: :default_project_visibility, form: f, selected_level: @application_setting.default_project_visibility, form_model: Project.new)
16: .form-group.visibility-level-setting
17: = f.label :default_snippet_visibility, class: ‘label-bold’
18: = render(‘shared/visibility_radios’, model_method: :default_snippet_visibility, form: f, selected_level: @application_setting.default_snippet_visibility, form_model: ProjectSnippet.new)
app/models/application_setting_implementation.rb:314:in pick_repository_storage' lib/gitlab/current_settings.rb:20:in method_missing’
app/models/concerns/has_repository.rb:23:in pick_repository_storage' app/models/project.rb:72:in block in class:Project’
app/models/project.rb:727:in initialize' app/views/admin/application_settings/_visibility_and_access.html.haml:15 app/views/admin/application_settings/_visibility_and_access.html.haml:1 app/views/admin/application_settings/general.html.haml:14 app/controllers/application_controller.rb:133:in render’
app/controllers/application_controller.rb:497:in set_current_admin' lib/gitlab/session.rb:11:in with_session’
app/controllers/application_controller.rb:488:in set_session_storage' lib/gitlab/i18n.rb:55:in with_locale’
lib/gitlab/i18n.rb:61:in with_user_locale' app/controllers/application_controller.rb:482:in set_locale’
lib/gitlab/error_tracking.rb:51:in with_context' app/controllers/application_controller.rb:547:in sentry_context’
app/controllers/application_controller.rb:475:in block in set_current_context' lib/gitlab/application_context.rb:52:in block in use’
lib/gitlab/application_context.rb:52:in use' lib/gitlab/application_context.rb:20:in with_context’
app/controllers/application_controller.rb:468:in set_current_context' lib/gitlab/request_profiler/middleware.rb:17:in call’
lib/gitlab/middleware/go.rb:20:in call' lib/gitlab/etag_caching/middleware.rb:13:in call’
lib/gitlab/middleware/multipart.rb:125:in call' lib/gitlab/middleware/read_only/controller.rb:51:in call’
lib/gitlab/middleware/read_only.rb:18:in call' lib/gitlab/middleware/same_site_cookies.rb:27:in call’
lib/gitlab/middleware/basic_health_check.rb:25:in call' lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in call’
lib/gitlab/middleware/request_context.rb:23:in call' config/initializers/fix_local_cache_middleware.rb:9:in call’
lib/gitlab/metrics/requests_rack_middleware.rb:60:in call' lib/gitlab/middleware/release_env.rb:12:in call’
Started GET “/-/metrics” for 127.0.0.1 at 2020-07-06 03:13:21 -0700
Processing by MetricsController#index as HTML
Completed 200 OK in 1ms (Views: 0.2ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms | Allocations: 307)
Started GET “/-/metrics” for 127.0.0.1 at 2020-07-06 03:13:36 -0700
Processing by MetricsController#index as HTML
Completed 200 OK in 1ms (Views: 0.2ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms | Allocations: 307)
Started GET “/-/metrics” for 127.0.0.1 at 2020-07-06 03:13:51 -0700
Processing by MetricsController#index as HTML
Completed 200 OK in 1ms (Views: 0.2ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms | Allocations: 307)
Started GET “/-/metrics” for 127.0.0.1 at 2020-07-06 03:14:06 -0700
Processing by MetricsController#index as HTML
Completed 200 OK in 1ms (Views: 0.3ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms | Allocations: 307)
Started GET “/” for 10.1.0.63 at 2020-07-06 03:14:11 -0700
Processing by RootController#index as HTML
Redirected to https://gitlab.guildmortgage.com/users/sign_in
Filter chain halted as :redirect_unlogged_user rendered or redirected
Completed 302 Found in 10ms (ActiveRecord: 0.0ms | Elasticsearch: 0.0ms | Allocations: 5402)
Started GET “/users/sign_in” for 10.1.0.63 at 2020-07-06 03:14:11 -0700
Processing by SessionsController#new as HTML
Completed 200 OK in 23ms (Views: 10.4ms | ActiveRecord: 0.9ms | Elasticsearch: 0.0ms | Allocations: 12373)
Started GET “/-/metrics” for 127.0.0.1 at 2020-07-06 03:14:21 -0700
Processing by MetricsController#index as HTML
Completed 200 OK in 1ms (Views: 0.2ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms | Allocations: 307)
Started GET “/-/metrics” for 127.0.0.1 at 2020-07-06 03:14:36 -0700
Processing by MetricsController#index as HTML
Completed 200 OK in 1ms (Views: 0.3ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms | Allocations: 307)