How to remove changed hosts keys from known_hosts

Greetings,

One of my servers was compromised (password reset vulnerability), so I erased the container image and rebuilt it using the latest release. This means that the host key will change.

It recover the repos, I wanted to use the repo push mechanism which had worked previously. But now, the pushing server has the wrong host key. I can see the error message in the mirror push, but I cannot find where the hosts keys are kept.

Is there some way to remove out-dated host SSH keys that gitlab would be using?

Cheers

P.S. The error message says where the known_hosts file is, but it’s not helpful because it refers to a temporary file which I suspect is being auto-generated from data in the database.

@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @\r\n@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@\r\nIT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!\r\nSomeone could be eavesdropping on you right now (man-in-the-middle attack)!\r\nIt is also possible that a host key has just been changed.\r\nThe fingerprint for the ECDSA key sent by the remote host is\nSHA256:DvsKf4gN6Uyg/uUdL1mDzCZG2rblSpvD9UmpiDzgKIY.\r\nPlease contact your system administrator.\r\nAdd correct host key in /tmp/gitaly-ssh-invocation1986799739/known-hosts to get rid of this message.\r\nOffending RSA key in /tmp/gitaly-ssh-invocation1986799739/known-hosts:3\r\n remove with:\r\n ssh-keygen -f "/tmp/gitaly-ssh-invocation1986799739/known-hosts" -R "[HOSTNAME]:PORT"\r\nECDSA host key for [HOSTNAME]:PORT has changed and you have requested strict checking.\r\nHost key verification failed.\r\nfatal: Could not read from remote repository.\n\nPlease make sure you have the correct access rights\nand the repository exists.\n".

I believe I figured out how to work around this issue. The host key is part of the mirroring data, so I needed to remove the mirroring and re-create it which will fetch a new host key. The mirroring works once it has been setup with the new host’s key.