GitHub Import without allowing GitHub SSO?

I currently have a GitLab instance that uses my company’s internal SSO (SAML based) and allows anyone with a valid SAML account access without approval.

I’m looking at what it would take to import projects from GitHub, and it looks like step one is to configure gitlab as an OAuth client to GitHub. But once I do this, users will also be able to login using GitHub, defeating the purpose of integrating with our internal SSO.

Is there a way to configure Git Hub OAuth for project importing only, without opening up logins via GitHub as well?

Did you ever find a solution to this?