After successful SAML authentication, self hosted GitLab gives a 500 error

I just upgraded to GitLab 15.0.2, and everything went well, including the reconfiguration. Afterwards, I tried to log in and was presented with a 500 error. I noticed that the SAML certificate was expired, so I renewed it, and put in the appropriate information. Reconfigured. Still got the 500 error. I then pulled down the new gitlab.rb SAML template, filled in the appropriate information, reconfigured, tried authenticating again. 500 error.

I then went GitLab’s SAML documentation to make sure that my configuration was correct. It said that the 500 error had something to do with the cert or fingerprint. I started messing around with them by adding, removing, changing the format of both the idp_cert_fingerprint and idp_cert arguments. I did things like leave the fingerprint with no colons (35AB56), added colons (35:AB:56), and even added spaces(35 AB 56). I’ve left the cert as a block, and removed all line breaks. I’ve used only one or the other, and even tried both. Reconfigured each time I made a change. All gave me 500 errors.

After some searching, I tailed the gitlab-ctl logs, and I found the following:

==> /var/log/gitlab/gitlab-rails/production.log <==

Gitlab::Auth::Ldap::Config::InvalidProvider (Unknown provider (ldapmain). Available providers: []):
  
lib/gitlab/auth/ldap/config.rb:62:in `invalid_provider'
lib/gitlab/auth/ldap/config.rb:73:in `initialize'
ee/app/models/ee/user.rb:442:in `new'
ee/app/models/ee/user.rb:442:in `block_auto_created_users?'
ee/app/models/ee/user.rb:413:in `blocked_auto_created_oauth_ldap_user?'
ee/app/models/ee/user.rb:407:in `activate_based_on_user_cap?'
ee/lib/ee/gitlab/auth/o_auth/user.rb:26:in `activate_user_based_on_user_cap?'
ee/lib/ee/gitlab/auth/o_auth/user.rb:11:in `activate_user_if_user_cap_not_reached'
lib/gitlab/auth/o_auth/user.rb:58:in `save'
ee/lib/ee/gitlab/auth/saml/user.rb:34:in `find_and_update!'
app/controllers/omniauth_callbacks_controller.rb:162:in `sign_in_user_flow'
app/controllers/omniauth_callbacks_controller.rb:130:in `omniauth_flow'
app/controllers/omniauth_callbacks_controller.rb:49:in `saml'
ee/lib/gitlab/ip_address_state.rb:10:in `with'
ee/app/controllers/ee/application_controller.rb:45:in `set_current_ip_address'
app/controllers/application_controller.rb:527:in `set_current_admin'
lib/gitlab/session.rb:11:in `with_session'
app/controllers/application_controller.rb:518:in `set_session_storage'
lib/gitlab/i18n.rb:105:in `with_locale'
lib/gitlab/i18n.rb:111:in `with_user_locale'
app/controllers/application_controller.rb:512:in `set_locale'
app/controllers/application_controller.rb:506:in `set_current_context'
lib/gitlab/middleware/memory_report.rb:13:in `call'
lib/gitlab/middleware/speedscope.rb:13:in `call'
lib/gitlab/database/load_balancing/rack_middleware.rb:23:in `call'
lib/gitlab/jira/middleware.rb:19:in `call'
lib/gitlab/middleware/go.rb:20:in `call'
lib/gitlab/etag_caching/middleware.rb:21:in `call'
lib/gitlab/middleware/query_analyzer.rb:11:in `block in call'
lib/gitlab/database/query_analyzer.rb:37:in `within'
lib/gitlab/middleware/query_analyzer.rb:11:in `call'
lib/gitlab/middleware/multipart.rb:173:in `call'
lib/gitlab/middleware/read_only/controller.rb:50:in `call'
lib/gitlab/middleware/read_only.rb:18:in `call'
lib/gitlab/middleware/same_site_cookies.rb:27:in `call'
lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'
lib/gitlab/middleware/basic_health_check.rb:25:in `call'
lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'
lib/gitlab/middleware/request_context.rb:21:in `call'
lib/gitlab/middleware/webhook_recursion_detection.rb:15:in `call'
config/initializers/fix_local_cache_middleware.rb:11:in `call'
lib/gitlab/middleware/compressed_json.rb:26:in `call'
lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call'
lib/gitlab/middleware/sidekiq_web_static.rb:20:in `call'
lib/gitlab/metrics/requests_rack_middleware.rb:77:in `call'
lib/gitlab/middleware/release_env.rb:13:in `call'

==> /var/log/gitlab/gitlab-workhorse/current <==
{"content_type":"text/html; charset=utf-8","correlation_id":"01G57MJ3ESQK6VVFS7JC14QCV5","duration_ms":259,"host":"{redacted}","level":"info","method":"POST","msg":"access","proto":"HTTP/1.1","referrer":"https://login.microsoftonline.com/","remote_addr":"127.0.0.1:0","remote_ip":"127.0.0.1","route":"","status":500,"system":"http","time":"2022-06-10T14:23:16-06:00","ttfb_ms":259,"uri":"/users/auth/saml/callback","user_agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.5 Safari/605.1.15","written_bytes":3049}

==> /var/log/gitlab/nginx/gitlab_access.log <==
10.96.250.6 - - [10/Jun/2022:14:23:16 -0600] "POST /users/auth/saml/callback HTTP/1.1" 500 3049 "https://login.microsoftonline.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.5 Safari/605.1.15" -

We had used LDAP at one time years ago, but have since moved on using Azure AD and SAML. I saw that LDAP was still configured. I then removed it from gitlab.rb, reconfigured. Tried again. 500 error.

I have tried all sorts of different variations that I have lost count. I have captured and inspected the SAML response that GitLab is getting, and it is valid with all the correct information. I know that I have a valid cert that is set to sign the response. I’m at a loss of why I keep on getting the 500 error. I only thing that I see in the logs that GitLab is still looking for LDAP even with LDAP not configured as I keep getting the message above in the logs.

How can I fix this 500 error? If it does have to do something with the LDAP error, how tell GitLab to not look for an LDAP provider without being able to log into GitLab?

Here’s my gitlab.rb file, with the sensitive stuff redacted:

## GitLab configuration settings
##! This file is generated during initial installation and **is not** modified
##! during upgrades.
##! Check out the latest version of this file to know about the different
##! settings that can be configured, when they were introduced and why:
##! https://gitlab.com/gitlab-org/omnibus-gitlab/blame/master/files/gitlab-config-template/gitlab.rb.template

##! Locally, the complete template corresponding to the installed version can be found at:
##! /opt/gitlab/etc/gitlab.rb.template

##! You can run `gitlab-ctl diff-config` to compare the contents of the current gitlab.rb with
##! the gitlab.rb.template from the currently running version.

##! You can run `gitlab-ctl show-config` to display the configuration that will be generated by
##! running `gitlab-ctl reconfigure`

##! In general, the values specified here should reflect what the default value of the attribute will be.
##! There are instances where this behavior is not possible or desired. For example, when providing passwords,
##! or connecting to third party services.
##! In those instances, we endeavour to provide an example configuration.

## GitLab URL
##! URL on which GitLab will be reachable.
##! For more details on configuring external_url see:
##! https://docs.gitlab.com/omnibus/settings/configuration.html#configuring-the-external-url-for-gitlab
##!
##! Note: During installation/upgrades, the value of the environment variable
##! EXTERNAL_URL will be used to populate/replace this value.
##! On AWS EC2 instances, we also attempt to fetch the public hostname/IP
##! address from AWS. For more details, see:
##! https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html
external_url '{gitlab_server_url}'
nginx['redirect_http_to_https'] = true

### OmniAuth Settings
###! Docs: https://docs.gitlab.com/ee/integration/omniauth.html
gitlab_rails['omniauth_enabled'] = true
gitlab_rails['omniauth_allow_single_sign_on'] = ['saml']
# gitlab_rails['omniauth_sync_email_from_provider'] = 'saml'
# gitlab_rails['omniauth_sync_profile_from_provider'] = ['saml']
# gitlab_rails['omniauth_sync_profile_attributes'] = ['email']
# gitlab_rails['omniauth_auto_sign_in_with_provider'] = 'saml'
gitlab_rails['omniauth_block_auto_created_users'] = false
gitlab_rails['omniauth_auto_link_ldap_user'] = true
gitlab_rails['omniauth_auto_link_saml_user'] = true
# gitlab_rails['omniauth_auto_link_user'] = ['saml']
# gitlab_rails['omniauth_external_providers'] = ['twitter', 'google_oauth2']
# gitlab_rails['omniauth_allow_bypass_two_factor'] = ['google_oauth2']
gitlab_rails['omniauth_providers'] = [
  {
    name: "saml",
    label: "Ensign College AD",
    args: { 
        assertion_consumer_service_url: "https://{gitlab_server}/users/auth/saml/callback",
        idp_cert_fingerprint: "{fingerprint}",
        idp_cert: "{saml_cert}",
        idp_sso_target_url: "{target_URL}",
        issuer: "https://{gitlab_server}/gitlab",
        name_identifier_format: "urn:oasis:names:tc:SAML:2.0:nameid-format:persistent",
        attribute_statements: { 
            email: ['http://schemas.xmlsoap.org/ws/2005/05/identity/claims/emailaddress'],
            nickname: ['http://schemas.xmlsoap.org/ws/2005/05/identity/claims/name'],
            name: ['http://schemas.microsoft.com/identity/claims/displayname']
        }
    }
  }
]


################################################################################
# Let's Encrypt integration
################################################################################
letsencrypt['enable'] = false
# letsencrypt['contact_emails'] = [] # This should be an array of email addresses to add as contacts
# letsencrypt['group'] = 'root'
# letsencrypt['key_size'] = 2048
# letsencrypt['owner'] = 'root'
# letsencrypt['wwwroot'] = '/var/opt/gitlab/nginx/www'
# See http://docs.gitlab.com/omnibus/settings/ssl.html#automatic-renewal for more on these sesttings
# letsencrypt['auto_renew'] = true
# letsencrypt['auto_renew_hour'] = 0
# letsencrypt['auto_renew_minute'] = nil # Should be a number or cron expression, if specified.
# letsencrypt['auto_renew_day_of_month'] = "*/4"

Similar problem here: After updating we also had “Unknown provider (ldapmain)” for some users, but not for all. After some digging, I found the problem:

When users first log in, GitLab stores the associated authenticator in its database. When you change your authenticator in gitlab.rb, this will not be reflected in the DB. Instead, a second association will be stored with the new authenticator. In our case, instead of just one LDAP server, we added several with different names. People who logged in before that change still had an entry with ‘ldapmain’ as provider.

You can investigate this in the Rails console by doing

 pp Identity.where(user_id: YOUR_USER_ID)

You should see two entries: on with ‘ldapmain’ as provider, and one with your SAML provider.

It seems that new GitLab versions blow up when you have that problem in your DB, while older versions seemed to ignore that. In our case, the solution was to delete all these old ‘ldapmain’ provider entries by doing

Identity.where(provider: 'ldapmain').each { |i| i.destroy }

Of course, have a backup ready before you do this. Also, be aware you might need to delete cookies before trying to authenticate again.

(See also: Admin user identities tab should gracefully handle missing/removed providers (#29342) · Issues · GitLab.org / GitLab FOSS · GitLab)

I’ve finally go around to trying your suggestions. And all I can say is ‘You da’ Man (or Woman)!’ Thank you so much!