Gitlab migrations and email-spam

I am working on the migration of a massive Redmine platform to Gitlab and got it mostly figured out.

I did the entire migration via the API, because there is not obvious supported migration path from Redmine.

I inserted issues with their historic dates and all, making use of impersonation tokens.
However, now I have the problem that the Gitlab instance is trying to send notification mails for all tickets that were created and all mentions, which is an insane volume and overwhelms the instances ability to send password-reset mails for a couple of days.

Here is what I have tried so far:

  • immediately after user creation I disabled notifications for all users using this endpoint:
    Notification settings API | GitLab
  • for all projects I have emails_disabled to true when creating them, see here:
    Notification settings API | GitLab
    after all issues and notes were created, I set emails_disabled to false
  • for groups I set emails_disabled and mentions_disabled to true, and then to false after migration
  • I have provided non-existing mails for all users, hoping Gitlab would exhaust itself at some point,
    but it kept retrying to send mails indefinitely

Now when I set user notification levels to “participating” again after migration, Gitlab spams mails.
(again, see here: Notification settings API | GitLab)

My current somewhat working approach is to just install a lightning-rod mail address and wait for the flood to end, but that is a really bad approach, because it makes a single migration a matter of days, instead of about an hour that the pure API-interaction takes at this point.

Figuring out how exactly we want to migrate needs some look-and-feel, hands-on experience from our prospective users, so I need to provide them with a migrated instance.
Gitlab is very unforgiving when trying to manipulate data directly in DB and I won’t try to manipulate the instance data via the API.
So I need to be able to set up and tear down instances in a somewhat “agile” fashion.

Ideally, I would like to prevent any notifications for events that happened in the past to be sent by Gitlab, given a point in time.
Or something else. Any idea would be greatly appreciated, this is really maiming the progress here.

(Talking about a self-hosted omnibus-instance btw)

Apparently the root issue is that during migration a large number of new_note and new_issue jobs are created in sidekiq, which in turn create mailer and mail_scheduler:mail_scheduler_notification_service jobs.

Fortunately. this API allows to kill jobs: Sidekiq queues administration API | GitLab

So the prime question now seems to be: how do I scale sidekiq in an omnibus installation?