Issue on gitlab omnibus: Getting a 500 error when updating Outbound Requests in Admin

Hello there,

We’re hosting the gitlab omnibus application in an AWS account and running version 13.9.1-ee. I’m trying to update the outbound requests to allow us to append an internal webhook to a project. When visiting the Admin Area → Setttings → Network → Outbound Requests, add an entry i.e example.com click save changes I get a http 500.

Checking the /var/log/gitlab/gitlab-rails/production_json.log on the server I get the below error message. It appears to be a OpenSSL::Cipher::CipherError type error, but very little diagnostic information is given.

Has anyone seen this before? If someone could respond that would be extremely helpful for us!

Thanks,
Matt

{
"method": "PATCH",
"path": "/admin/application_settings/network",
"format": "html",
"controller": "Admin::ApplicationSettingsController",
"action": "network",
"status": 500,
"time": "2021-03-17T12:25:34.764Z",
"params": [
    {
        "key": "utf8",
        "value": "✓"
    },
    {
        "key": "_method",
        "value": "patch"
    },
    {
        "key": "authenticity_token",
        "value": "[FILTERED]"
    },
    {
        "key": "application_setting",
        "value": {
            "allow_local_requests_from_web_hooks_and_services": "1",
            "allow_local_requests_from_system_hooks": "1",
            "outbound_local_requests_allowlist_raw": "example.com",
            "dns_rebinding_protection_enabled": "1"
        }
    }
],
"remote_ip": "XXX",
"user_id": 14,
"username": "matt.gawne",
"ua": "Mozilla/5.0(Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36",
"correlation_id": "01F102BZ7Z3GWVT2EKWMWA6BKA",
"meta.user": "matt.gawne",
"meta.caller_id": "Admin::ApplicationSettingsController#network",
"meta.remote_ip": "XXX",
"meta.feature_category": "not_owned",
"redis_calls": 6,
"redis_duration_s": 0.001715,
"redis_read_bytes": 1031,
"redis_write_bytes": 1122,
"redis_cache_calls": 5,
"redis_cache_duration_s": 0.001346,
"redis_cache_read_bytes": 848,
"redis_cache_write_bytes": 247,
"redis_shared_state_calls": 1,
"redis_shared_state_duration_s": 0.000369,
"redis_shared_state_read_bytes": 183,
"redis_shared_state_write_bytes": 875,
"db_count": 30,
"db_write_count": 0,
"db_cached_count": 25,
"cpu_s": 0.038537,
"queue_duration_s": 0.010149,
"exception.class": "OpenSSL::Cipher::CipherError",
"exception.message": "",
"exception.backtrace": [
    "app/services/application_settings/update_service.rb:50:in `update_settings'",
    "lib/gitlab/metrics/instrumentation.rb:160:in `block in update_settings'",
    "lib/gitlab/metrics/method_call.rb:27:in `measure'",
    "lib/gitlab/metrics/instrumentation.rb:160:in `update_settings'",
    "app/services/application_settings/update_service.rb:12:in `execute'",
    "ee/app/services/ee/application_settings/update_service.rb:22:in `execute'",
    "app/controllers/admin/application_settings_controller.rb:263:in `perform_update'",
    "app/controllers/admin/application_settings_controller.rb:50:in `block (2 levels) in <class:ApplicationSettingsController>'",
    "ee/lib/gitlab/ip_address_state.rb:10:in `with'",
    "ee/app/controllers/ee/application_controller.rb:44:in `set_current_ip_address'",
    "app/controllers/application_controller.rb:482:in `set_current_admin'",
    "lib/gitlab/session.rb:11:in `with_session'",
    "app/controllers/application_controller.rb:473:in `set_session_storage'",
    "lib/gitlab/i18n.rb:73:in `with_locale'",
    "lib/gitlab/i18n.rb:79:in `with_user_locale'",
    "app/controllers/application_controller.rb:467:in `set_locale'",
    "lib/gitlab/error_tracking.rb:52:in `with_context'",
    "app/controllers/application_controller.rb:532:in `sentry_context'",
    "app/controllers/application_controller.rb:460:in `block in set_current_context'",
    "lib/gitlab/application_context.rb:56:in `block in use'",
    "lib/gitlab/application_context.rb:56:in `use'",
    "lib/gitlab/application_context.rb:22:in `with_context'",
    "app/controllers/application_controller.rb:451:in `set_current_context'",
    "lib/gitlab/metrics/elasticsearch_rack_middleware.rb:16:in `call'",
    "lib/gitlab/middleware/rails_queue_duration.rb:33:in `call'",
    "lib/gitlab/metrics/rack_middleware.rb:16:in `block in call'",
    "lib/gitlab/metrics/transaction.rb:56:in `run'",
    "lib/gitlab/metrics/rack_middleware.rb:16:in `call'",
    "lib/gitlab/request_profiler/middleware.rb:17:in `call'",
    "lib/gitlab/jira/middleware.rb:19:in `call'",
    "lib/gitlab/middleware/go.rb:20:in `call'",
    "lib/gitlab/etag_caching/middleware.rb:21:in `call'",
    "lib/gitlab/middleware/multipart.rb:172:in `call'",
    "lib/gitlab/middleware/read_only/controller.rb:50:in `call'",
    "lib/gitlab/middleware/read_only.rb:18:in `call'",
    "lib/gitlab/middleware/same_site_cookies.rb:27:in `call'",
    "lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'",
    "lib/gitlab/middleware/basic_health_check.rb:25:in `call'",
    "lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'",
    "lib/gitlab/middleware/request_context.rb:21:in `call'",
    "config/initializers/fix_local_cache_middleware.rb:11:in `call'",
    "lib/gitlab/metrics/requests_rack_middleware.rb:76:in `call'",
    "lib/gitlab/middleware/release_env.rb:12:in `call'"
],
"db_duration_s": 0.0033,
"view_duration_s": 0.0,
"duration_s": 0.03183

}

Hi,

Not sure I will be able to help much, I’ve just checked on my install to check/verify if there is a bigger issue or not. Unfortunately (or fortunately), it did work for me, so I wasn’t able to replicate the problem. I was able to add a domain to the box below, as well as enable/disable the option for web hooks.

The only thing I can suggest, is check/verify that in /etc/gitlab/gitlab.rb to ensure all your certificate configuration is OK. I have two servers, one using letsencrypt, and the other (where I just tested), using self-generated certs with an internal CA that I run in-house without external access. This is the only thing I could think of that you might want to check considering the CipherErrors in your log output.

For example, mine is commented, so I’m using the defaults:

# nginx['ssl_ciphers'] = "ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256"
# postgresql['ssl_ciphers'] = 'HIGH:MEDIUM:+3DES:!aNULL:!SSLv3:!TLSv1'

perhaps something was changed here that is causing you issues - you could revert to the default config as per mine above and see if it works. My postgresql ssl_ciphers are also using the defaults. So you may wish to check/verify these as well as ensuring the certificate that you are using with Gitlab is also correctly configured and not causing any conflicts. Of course, after making changes, gitlab-ctl reconfigure and gitlab-ctl restart as well.

My only other difference is version 13.9.4 in comparison to yours but that isn’t likely to be the issue, but more a config problem with too restrictive ciphers perhaps.

Hi iwalker,

Thank you for taking the time to get back to me so quickly after I posted.

Glad to hear it isn’t an issue across the board. I’ve checked the gitlab.rb files on our gitlab configured instances (we have three of them at the front end) and they all have letsencypt set to false and have the nginx and postresql blocks (above) commented unfortunately.

I haven’t tried updating the boxes yet but it looks like this could be the solution.

Matt