Email On Push Keeps Sending

We enabled Emails on push for all projects but have a serious issue with it. It won’t stop sending emails!

Our gitlab is self managed core, 12.0.0. The emails are sent through gmail. All normal email notifications are working fine, but these ones are sending hundreds at a time. First thing we did was deactivate it, but that hasn’t stopped it. I’ve changed all the settings to, to different email addresses, but it’s still sending to the original emails added before.

If I clear out the jobs it stops for a while, but then the next time we push or tag, it’ll queue up hundreds to send again.

Has anyone else had this problem or know any way to clear out these settings to stop it happening again?