Cannot remove access of a user as owner of git repo

I am listed as Owner of git repository, and I want to remove an inactive user from accessing the repo. I can remove a user that I have added after becoming an owner, but I cannot remove a user who joined repo before I was transferred the Owner rights. Is there a way to remove this inactive user, or any workarounds, besides setting up a new account and migrating the repo?