No rollback buttons?

When looking at a specific Environment, I seem to remember having a “Rollback” button that would execute the associated job for an earlier commit. But I can’t seem to find it anymore. I am running Community version 11.5.4.

Are there any permissions/settings I am unaware of? I am a Maintainer on the project and even tried to disable protection on the branch.

Edit:
It seems to work for new commits/jobs. Is there a timeout on a month or something?