How to remove <legacy> gitlab-runner after upgrade?

I recently upgraded my gitlab-runner from 14.7.0 to 15.11.0 and I notice that sometimes a task is picked up by the old 14.7.0 runner. How is this possible when the binary has been replaced?
In the UI I see a symbol which indicates that there are two runners in the group. Clicking on a runner shows me the details and lists a 14.7.0 legacy runner. It’s breaking my pipelines and I need to get rid of it. Suggestions?