Rules - skip if changes only to specific file

With this:

.ignore-readme:
  rules:
    - changes:
        - "README.md"
      when: never
    - when: always

my pipeline is not triggered if there are any changes to README.md regardless of whether there are changes in other files.

I want it to work like this:
If the only file with changes in a commit is README.md - don’t trigger. (Works as intended)
If any other file has changes - trigger. (Works as intended)
If any other file AND README.md - trigger. (Right now it doesn’t)

How can I fix this?