CI pipeline runs on normal push despite 'only: merge_requests' tag

I only want my CI pipeline to run on merge requests, not on every push. I have only: - merge_requests on the single job in my yml file. Still, the pipeline is run if I push to gitlab, without any merge request involved. What am I missing?