How to clean database after rogue front-end

We accidentally added a new “front-end” to our Gitlab CE server with the wrong gitlab-secrets.json file.

During the 36 hours that this persisted, the unicorn workers processed a number of new user jobs. Now we have some bad tokens in the database and affected users get a 500 error for many operations. Also, the Admin interface gets the same errors, so we can’t delete bad jobs/repos through the web interface.

We can clean these on a case by case basis using an admin-access-only front-end with:

  • gitlab-ctl stop
  • cp bad-gitlab-secrets.json /etc/gitlab/gitlab-secrets.json
  • gitlab-ctl reconfigure
  • gitlab-ctl start
    use the APIs and/or the web interface to delete jobs/repositories/whatever, then, as soon as possible:
  • gitlab-ctl stop
  • cp good.gitlab-secrets.json /etc/gitlab/gitlab-secrets.json
  • gitlab-ctl reconfigure
  • gitlab-ctl start

However, during the several minutes that the server is running with the bad secrets, it can and sometimes does pick up more jobs from the queue and create more pollution in the database.

Is there a way to find and remove any objects in the database that have the wrong secrets attached?

Many thanks for any tips