Openid connect not working after 17.1.1

Problem to solve

if i try to login with openid connect (keycloak → microsoft → gitlab)

500
We're sorry. Something went wrong on our end.
Request ID: `01J2BH8B27CZGK2Z71B55HXBPZ`

it was working for month until i upgrade to 17.1.1 yesterday

Configuration

gitlab_rails['omniauth_enabled'] = true
gitlab_rails['omniauth_providers'] = [
  {
    "name" => "keycloak",
    "label" => "Keycloak SSO",
    "args" => {
      "name" => "openid_connect", "strategy_class": "OmniAuth::Strategies::OpenIDConnect",
      "scope" => ["openid", "profile", "email"],
      "response_type" => "code",
      "issuer" => "https://keycloak.*****.it/realms/Credenziali",
      "client_auth_method" => "query",
      "uid_field" => "uid",
      "send_scope_to_token_endpoint" => "false",
      "pkce" => "true",
      "discovery" => true,
      "client_options" => {
        "identifier" => "gitlab.****.it",
        "secret" => "*******************************",
        "redirect_uri" => "https://gitlab.******.it/users/auth/openid_connect/callback"
      }
    }
  }
]


gitlab_rails['omniauth_auto_link_user'] = ['openid_connect']

Versions

Please select whether options apply, and add the version information.

  • [ * ] Self-managed

Versions

System information
System: Ubuntu 22.04
Proxy: no
Current User: git
Using RVM: no
Ruby Version: 3.1.5p253
Gem Version: 3.5.11
Bundler Version:2.5.11
Rake Version: 13.0.6
Redis Version: 7.0.15
Sidekiq Version:7.1.6
Go Version: unknown

GitLab information
Version: 17.1.1-ee
Revision: d0ac56e0be2
Directory: /opt/gitlab/embedded/service/gitlab-rails
DB Adapter: PostgreSQL
DB Version: 14.11
URL: https://gitlab..it
HTTP Clone URL: https://gitlab.
.it/some-group/some-project.git
SSH Clone URL: git@gitlab.******.it:some-group/some-project.git
Elasticsearch: no
Geo: no
Using LDAP: yes
Using Omniauth: yes
Omniauth Providers: keycloak

GitLab Shell
Version: 14.36.0
Repository storages:

  • default: unix:/var/opt/gitlab/gitaly/gitaly.socket
    GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell

Gitaly

  • default Address: unix:/var/opt/gitlab/gitaly/gitaly.socket
  • default Version: 17.1.1
  • default Git Version: 2.45.1

relevant log:

{
    "action": "openid_connect",
    "controller": "OmniauthCallbacksController",
    "correlation_id": "01J2BH8B27CZGK2Z71B55HXBPZ",
    "cpu_s": 0.113558,
    "db_cached_count": 0,
    "db_ci_cached_count": 0,
    "db_ci_count": 0,
    "db_ci_duration_s": 0.0,
    "db_ci_replica_cached_count": 0,
    "db_ci_replica_count": 0,
    "db_ci_replica_duration_s": 0.0,
    "db_ci_replica_txn_count": 0,
    "db_ci_replica_txn_duration_s": 0.0,
    "db_ci_replica_txn_max_duration_s": 0.0,
    "db_ci_replica_wal_cached_count": 0,
    "db_ci_replica_wal_count": 0,
    "db_ci_txn_count": 0,
    "db_ci_txn_duration_s": 0.0,
    "db_ci_txn_max_duration_s": 0.0,
    "db_ci_wal_cached_count": 0,
    "db_ci_wal_count": 0,
    "db_count": 1,
    "db_duration_s": 0.00095,
    "db_main_cached_count": 0,
    "db_main_count": 1,
    "db_main_duration_s": 0.001,
    "db_main_replica_cached_count": 0,
    "db_main_replica_count": 0,
    "db_main_replica_duration_s": 0.0,
    "db_main_replica_txn_count": 0,
    "db_main_replica_txn_duration_s": 0.0,
    "db_main_replica_txn_max_duration_s": 0.0,
    "db_main_replica_wal_cached_count": 0,
    "db_main_replica_wal_count": 0,
    "db_main_txn_count": 0,
    "db_main_txn_duration_s": 0.0,
    "db_main_txn_max_duration_s": 0.0,
    "db_main_wal_cached_count": 0,
    "db_main_wal_count": 0,
    "db_primary_cached_count": 0,
    "db_primary_count": 1,
    "db_primary_duration_s": 0.001,
    "db_primary_txn_count": 0,
    "db_primary_txn_duration_s": 0.0,
    "db_primary_txn_max_duration_s": 0.0,
    "db_primary_wal_cached_count": 0,
    "db_primary_wal_count": 0,
    "db_replica_cached_count": 0,
    "db_replica_count": 0,
    "db_replica_duration_s": 0.0,
    "db_replica_txn_count": 0,
    "db_replica_txn_duration_s": 0.0,
    "db_replica_txn_max_duration_s": 0.0,
    "db_replica_wal_cached_count": 0,
    "db_replica_wal_count": 0,
    "db_txn_count": 0,
    "db_write_count": 0,
    "duration_s": 0.01746,
    "exception.backtrace": [
        "ee/lib/gitlab/auth/oidc/config.rb:22:in `required_groups'",
        "ee/lib/gitlab/auth/oidc/user.rb:42:in `required_groups_enabled?'",
        "ee/lib/gitlab/auth/oidc/user.rb:20:in `find_user'",
        "lib/gitlab/auth/o_auth/user.rb:75:in `gl_user'",
        "lib/gitlab/auth/o_auth/user.rb:261:in `update_profile'",
        "lib/gitlab/auth/o_auth/user.rb:34:in `initialize'",
        "ee/lib/gitlab/auth/oidc/user.rb:16:in `initialize'",
        "app/controllers/omniauth_callbacks_controller.rb:201:in `new'",
        "app/controllers/omniauth_callbacks_controller.rb:201:in `block in build_auth_user'",
        "gems/gitlab-utils/lib/gitlab/utils/strong_memoize.rb:65:in `strong_memoize_with'",
        "app/controllers/omniauth_callbacks_controller.rb:200:in `build_auth_user'",
        "app/controllers/omniauth_callbacks_controller.rb:209:in `sign_in_user_flow'",
        "app/controllers/omniauth_callbacks_controller.rb:168:in `omniauth_flow'",
        "ee/app/controllers/ee/omniauth_callbacks_controller.rb:15:in `openid_connect'",
        "actionpack (7.0.8.4) lib/action_controller/metal/basic_implicit_render.rb:6:in `send_action'",
        "actionpack (7.0.8.4) lib/abstract_controller/base.rb:215:in `process_action'",
        "actionpack (7.0.8.4) lib/action_controller/metal/rendering.rb:165:in `process_action'",
        "actionpack (7.0.8.4) lib/abstract_controller/callbacks.rb:234:in `block in process_action'",
        "activesupport (7.0.8.4) lib/active_support/callbacks.rb:118:in `block in run_callbacks'",
        "lib/gitlab/ip_address_state.rb:11:in `with'",
        "ee/app/controllers/ee/application_controller.rb:45:in `set_current_ip_address'",
        "activesupport (7.0.8.4) lib/active_support/callbacks.rb:127:in `block in run_callbacks'",
        "app/controllers/application_controller.rb:468:in `set_current_admin'",
        "activesupport (7.0.8.4) lib/active_support/callbacks.rb:127:in `block in run_callbacks'",
        "lib/gitlab/session.rb:11:in `with_session'",
        "app/controllers/application_controller.rb:459:in `set_session_storage'",
        "activesupport (7.0.8.4) lib/active_support/callbacks.rb:127:in `block in run_callbacks'",
        "lib/gitlab/i18n.rb:114:in `with_locale'",
        "app/controllers/application_controller.rb:452:in `set_locale'",
        "activesupport (7.0.8.4) lib/active_support/callbacks.rb:127:in `block in run_callbacks'",
        "marginalia (1.11.1) lib/marginalia.rb:109:in `record_query_comment'",
        "activesupport (7.0.8.4) lib/active_support/callbacks.rb:127:in `block in run_callbacks'",
        "app/controllers/application_controller.rb:443:in `set_current_context'",
        "activesupport (7.0.8.4) lib/active_support/callbacks.rb:127:in `block in run_callbacks'",
        "sentry-rails (5.17.3) lib/sentry/rails/controller_transaction.rb:28:in `block in sentry_around_action'",
        "sentry-ruby (5.17.3) lib/sentry/hub.rb:102:in `with_child_span'",
        "sentry-ruby (5.17.3) lib/sentry-ruby.rb:490:in `with_child_span'",
        "sentry-rails (5.17.3) lib/sentry/rails/controller_transaction.rb:14:in `sentry_around_action'",
        "activesupport (7.0.8.4) lib/active_support/callbacks.rb:127:in `block in run_callbacks'",
        "activesupport (7.0.8.4) lib/active_support/callbacks.rb:138:in `run_callbacks'",
        "actionpack (7.0.8.4) lib/abstract_controller/callbacks.rb:233:in `process_action'",
        "actionpack (7.0.8.4) lib/action_controller/metal/rescue.rb:23:in `process_action'",
        "actionpack (7.0.8.4) lib/action_controller/metal/instrumentation.rb:67:in `block in process_action'",
        "activesupport (7.0.8.4) lib/active_support/notifications.rb:206:in `block in instrument'",
        "activesupport (7.0.8.4) lib/active_support/notifications/instrumenter.rb:24:in `instrument'",
        "activesupport (7.0.8.4) lib/active_support/notifications.rb:206:in `instrument'",
        "actionpack (7.0.8.4) lib/action_controller/metal/instrumentation.rb:66:in `process_action'",
        "actionpack (7.0.8.4) lib/action_controller/metal/params_wrapper.rb:259:in `process_action'",
        "activerecord (7.0.8.4) lib/active_record/railties/controller_runtime.rb:27:in `process_action'",
        "actionpack (7.0.8.4) lib/abstract_controller/base.rb:151:in `process'",
        "actionview (7.0.8.4) lib/action_view/rendering.rb:39:in `process'",
        "actionpack (7.0.8.4) lib/action_controller/metal.rb:188:in `dispatch'",
        "actionpack (7.0.8.4) lib/action_controller/metal.rb:251:in `dispatch'",
        "actionpack (7.0.8.4) lib/action_dispatch/routing/route_set.rb:49:in `dispatch'",
        "actionpack (7.0.8.4) lib/action_dispatch/routing/route_set.rb:32:in `serve'",
        "actionpack (7.0.8.4) lib/action_dispatch/routing/mapper.rb:18:in `block in <class:Constraints>'",
        "actionpack (7.0.8.4) lib/action_dispatch/routing/mapper.rb:48:in `serve'",
        "actionpack (7.0.8.4) lib/action_dispatch/journey/router.rb:50:in `block in serve'",
        "actionpack (7.0.8.4) lib/action_dispatch/journey/router.rb:32:in `each'",
        "actionpack (7.0.8.4) lib/action_dispatch/journey/router.rb:32:in `serve'",
        "actionpack (7.0.8.4) lib/action_dispatch/routing/route_set.rb:852:in `call'",
        "gitlab-experiment (0.9.1) lib/gitlab/experiment/middleware.rb:19:in `call'",
        "omniauth (2.1.0) lib/omniauth/strategy.rb:470:in `call_app!'",
        "omniauth (2.1.0) lib/omniauth/strategy.rb:418:in `callback_phase'",
        "omniauth_openid_connect (0.6.1) lib/omniauth/strategies/openid_connect.rb:138:in `callback_phase'",
        "omniauth (2.1.0) lib/omniauth/strategy.rb:272:in `callback_call'",
        "omniauth (2.1.0) lib/omniauth/strategy.rb:194:in `call!'",
        "omniauth (2.1.0) lib/omniauth/strategy.rb:169:in `call'",
        "omniauth (2.1.0) lib/omniauth/strategy.rb:202:in `call!'",
        "omniauth (2.1.0) lib/omniauth/strategy.rb:169:in `call'",
        "flipper (0.26.2) lib/flipper/middleware/memoizer.rb:72:in `memoized_call'",
        "flipper (0.26.2) lib/flipper/middleware/memoizer.rb:37:in `call'",
        "lib/gitlab/middleware/sidekiq_shard_awareness_validation.rb:20:in `block in call'",
        "lib/gitlab/sidekiq_sharding/validator.rb:42:in `enabled'",
        "lib/gitlab/middleware/sidekiq_shard_awareness_validation.rb:20:in `call'",
        "lib/gitlab/middleware/memory_report.rb:13:in `call'",
        "lib/gitlab/middleware/speedscope.rb:13:in `call'",
        "lib/gitlab/database/load_balancing/rack_middleware.rb:23:in `call'",
        "lib/gitlab/middleware/go.rb:20:in `call'",
        "lib/gitlab/etag_caching/middleware.rb:21:in `call'",
        "lib/gitlab/middleware/query_analyzer.rb:11:in `block in call'",
        "lib/gitlab/database/query_analyzer.rb:40:in `within'",
        "lib/gitlab/middleware/query_analyzer.rb:11:in `call'",
        "lib/gitlab/middleware/organizations/current.rb:20:in `call'",
        "batch-loader (2.0.5) lib/batch_loader/middleware.rb:11:in `call'",
        "rack-attack (6.7.0) lib/rack/attack.rb:103:in `call'",
        "apollo_upload_server (2.1.6) lib/apollo_upload_server/middleware.rb:19:in `call'",
        "lib/gitlab/middleware/multipart.rb:173:in `call'",
        "rack-attack (6.7.0) lib/rack/attack.rb:127:in `call'",
        "warden (1.2.9) lib/warden/manager.rb:36:in `block in call'",
        "warden (1.2.9) lib/warden/manager.rb:34:in `catch'",
        "warden (1.2.9) lib/warden/manager.rb:34:in `call'",
        "rack-cors (2.0.1) lib/rack/cors.rb:102:in `call'",
        "rack (2.2.8.1) lib/rack/tempfile_reaper.rb:15:in `call'",
        "rack (2.2.8.1) lib/rack/etag.rb:27:in `call'",
        "rack (2.2.8.1) lib/rack/conditional_get.rb:27:in `call'",
        "rack (2.2.8.1) lib/rack/head.rb:12:in `call'",
        "actionpack (7.0.8.4) lib/action_dispatch/http/permissions_policy.rb:38:in `call'",
        "actionpack (7.0.8.4) lib/action_dispatch/http/content_security_policy.rb:36:in `call'",
        "lib/gitlab/middleware/read_only/controller.rb:50:in `call'",
        "lib/gitlab/middleware/read_only.rb:18:in `call'",
        "lib/gitlab/middleware/unauthenticated_session_expiry.rb:18:in `call'",
        "rack (2.2.8.1) lib/rack/session/abstract/id.rb:266:in `context'",
        "rack (2.2.8.1) lib/rack/session/abstract/id.rb:260:in `call'",
        "actionpack (7.0.8.4) lib/action_dispatch/middleware/cookies.rb:704:in `call'",
        "lib/gitlab/middleware/same_site_cookies.rb:27:in `call'",
        "actionpack (7.0.8.4) lib/action_dispatch/middleware/callbacks.rb:27:in `block in call'",
        "activesupport (7.0.8.4) lib/active_support/callbacks.rb:99:in `run_callbacks'",
        "actionpack (7.0.8.4) lib/action_dispatch/middleware/callbacks.rb:26:in `call'",
        "sentry-rails (5.17.3) lib/sentry/rails/rescued_exception_interceptor.rb:12:in `call'",
        "actionpack (7.0.8.4) lib/action_dispatch/middleware/debug_exceptions.rb:28:in `call'",
        "lib/gitlab/middleware/path_traversal_check.rb:27:in `call'",
        "lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'",
        "sentry-ruby (5.17.3) lib/sentry/rack/capture_exceptions.rb:29:in `block (2 levels) in call'",
        "sentry-ruby (5.17.3) lib/sentry/hub.rb:251:in `with_session_tracking'",
        "sentry-ruby (5.17.3) lib/sentry-ruby.rb:403:in `with_session_tracking'",
        "sentry-ruby (5.17.3) lib/sentry/rack/capture_exceptions.rb:20:in `block in call'",
        "sentry-ruby (5.17.3) lib/sentry/hub.rb:59:in `with_scope'",
        "sentry-ruby (5.17.3) lib/sentry-ruby.rb:383:in `with_scope'",
        "sentry-ruby (5.17.3) lib/sentry/rack/capture_exceptions.rb:19:in `call'",
        "actionpack (7.0.8.4) lib/action_dispatch/middleware/show_exceptions.rb:29:in `call'",
        "lib/gitlab/middleware/basic_health_check.rb:25:in `call'",
        "lograge (0.11.2) lib/lograge/rails_ext/rack/logger.rb:15:in `call_app'",
        "railties (7.0.8.4) lib/rails/rack/logger.rb:25:in `block in call'",
        "activesupport (7.0.8.4) lib/active_support/tagged_logging.rb:99:in `block in tagged'",
        "activesupport (7.0.8.4) lib/active_support/tagged_logging.rb:37:in `tagged'",
        "activesupport (7.0.8.4) lib/active_support/tagged_logging.rb:99:in `tagged'",
        "railties (7.0.8.4) lib/rails/rack/logger.rb:25:in `call'",
        "actionpack (7.0.8.4) lib/action_dispatch/middleware/remote_ip.rb:93:in `call'",
        "lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'",
        "lib/gitlab/middleware/request_context.rb:15:in `call'",
        "lib/gitlab/middleware/webhook_recursion_detection.rb:15:in `call'",
        "request_store (1.5.1) lib/request_store/middleware.rb:19:in `call'",
        "rack (2.2.8.1) lib/rack/method_override.rb:24:in `call'",
        "rack (2.2.8.1) lib/rack/runtime.rb:22:in `call'",
        "rack-timeout (0.6.3) lib/rack/timeout/core.rb:148:in `block in call'",
        "rack-timeout (0.6.3) lib/rack/timeout/support/timeout.rb:19:in `timeout'",
        "rack-timeout (0.6.3) lib/rack/timeout/core.rb:147:in `call'",
        "config/initializers/fix_local_cache_middleware.rb:11:in `call'",
        "lib/gitlab/middleware/compressed_json.rb:44:in `call'",
        "actionpack (7.0.8.4) lib/action_dispatch/middleware/executor.rb:14:in `call'",
        "lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call'",
        "rack (2.2.8.1) lib/rack/sendfile.rb:110:in `call'",
        "lib/gitlab/middleware/sidekiq_web_static.rb:20:in `call'",
        "lib/gitlab/metrics/requests_rack_middleware.rb:79:in `call'",
        "gitlab-labkit (0.36.0) lib/labkit/middleware/rack.rb:22:in `block in call'",
        "gitlab-labkit (0.36.0) lib/labkit/context.rb:35:in `with_context'",
        "gitlab-labkit (0.36.0) lib/labkit/middleware/rack.rb:21:in `call'",
        "actionpack (7.0.8.4) lib/action_dispatch/middleware/request_id.rb:26:in `call'",
        "actionpack (7.0.8.4) lib/action_dispatch/middleware/host_authorization.rb:131:in `call'",
        "railties (7.0.8.4) lib/rails/engine.rb:530:in `call'",
        "railties (7.0.8.4) lib/rails/railtie.rb:226:in `public_send'",
        "railties (7.0.8.4) lib/rails/railtie.rb:226:in `method_missing'",
        "lib/gitlab/middleware/release_env.rb:13:in `call'",
        "rack (2.2.8.1) lib/rack/urlmap.rb:74:in `block in call'",
        "rack (2.2.8.1) lib/rack/urlmap.rb:58:in `each'",
        "rack (2.2.8.1) lib/rack/urlmap.rb:58:in `call'",
        "puma (6.4.0) lib/puma/configuration.rb:272:in `call'",
        "puma (6.4.0) lib/puma/request.rb:100:in `block in handle_request'",
        "puma (6.4.0) lib/puma/thread_pool.rb:378:in `with_force_shutdown'",
        "puma (6.4.0) lib/puma/request.rb:99:in `handle_request'",
        "puma (6.4.0) lib/puma/server.rb:443:in `process_client'",
        "puma (6.4.0) lib/puma/server.rb:241:in `block in run'",
        "puma (6.4.0) lib/puma/thread_pool.rb:155:in `block in spawn_thread'"
    ],
    "exception.class": "NoMethodError",
    "exception.message": "undefined method `dig' for nil:NilClass\n\n          options.dig('args', 'client_options', 'gitlab', 'required_groups') || []\n                 ^^^^",
    "format": "html",
    "mem_bytes": 2835977,
    "mem_mallocs": 7111,
    "mem_objects": 22471,
    "mem_total_bytes": 3734817,
    "meta.caller_id": "OmniauthCallbacksController#openid_connect",
    "meta.client_id": "ip/192.168.8.7",
    "meta.feature_category": "system_access",
    "meta.remote_ip": "192.168.8.7",
    "method": "GET",
    "params": [
        {
            "key": "state",
            "value": "ab0eb8d7f39b6b3f40a1a9c7af02870c"
        },
        {
            "key": "session_state",
            "value": "c2895cf2-29b2-483d-9c1c-876d8f273860"
        },
        {
            "key": "iss",
            "value": "https://keycloak.*********.it/realms/Credenziali"
        },
        {
            "key": "code",
            "value": "[FILTERED]"
        }
    ],
    "path": "/users/auth/openid_connect/callback",
    "pid": 51758,
    "rate_limiting_gates": [
    ],
    "redis_calls": 6,
    "redis_duration_s": 0.001319,
    "redis_feature_flag_calls": 4,
    "redis_feature_flag_duration_s": 0.000744,
    "redis_feature_flag_read_bytes": 714,
    "redis_feature_flag_write_bytes": 203,
    "redis_read_bytes": 1163,
    "redis_sessions_calls": 2,
    "redis_sessions_duration_s": 0.000575,
    "redis_sessions_read_bytes": 449,
    "redis_sessions_write_bytes": 85,
    "redis_write_bytes": 288,
    "remote_ip": "192.168.8.7",
    "request_urgency": "default",
    "status": 500,
    "target_duration_s": 1,
    "time": "2024-07-09T10:40:01.065Z",
    "ua": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36",
    "view_duration_s": 0.0,
    "worker_id": "puma_0"
}

It seems to me that i have a similar case of not being able to use the OIDC login after upgrading GitLab.
Do you run GitLab with docker compose? If yes, did you upgrade the docker compose version (e.g. from v1 to v2) previously before the error occurred?

hi,
Gitlab is not running with docker
i had to roll back to a previus version to make it work again,
I haven’t had a chance to test further

Thanks for the reply.
I found out that I had a firewall issue, since changing the docker compose version from v1 to v2 some internal docker ip changed and my firewall rules didn’t work completely.

Hi,
i was able to upgrade to 17.3.1 without any issue now, maybe something didn’t go right the first time or there was a specific issue on that version