This project was scheduled for deletion, but failed with the following message: 7:permission denied

Hi, I have deployed gitlab in kubernetes by helm, the gitlab version is gitlab-8.6.1 an the helm version is v17.6.1 every thin works fine for instance, I can create projects push and pull things to them unless when trying to delete project it wont be deleted and I get this error:

This project was scheduled for deletion, but failed with the following message: 7:permission denied: token has expired. debug_error_string:{UNKNOWN:Error received from peer {created_time:"2024-12-10T10:17:37.739631219+00:00", grpc_status:7, grpc_message:"permission denied: token has expired"}}
The project visibility may have been made more restrictive if the parent group's visibility changed while the deletion was scheduled.

any idea what the problem is?

After many efforts and searching, we finally found out that GitLab pods that were running on different workers had different times and were not synced as their worker times were different than each other setting NTP on workers solved the problem for more information you can read this: