Today I’ve stumbled upon the strange problem, after kubernetes cluster restart I was unable to connect using kubernates operations, I’ve got an error in my job:
Error: Kubernetes cluster unreachable: the server has asked for the client to provide credentials
33error: You must be logged in to the server (Unauthorized)
This was an error when doing kubectl operation from the script, but the namespace was not created, so I assumed the configuration is broken and I’ve deleted the configuration and created it again, after that everything worked.
But I’ve inspected it more carefully and stated, that the gitlab CI is unable to recreate the namespace, if it was deleted.
We are using dynamic environments, specified by ENV variable, so that when starting the pipeline, we can deploy to the other stage. Sometimes we need more stages, that need to be cleaned up. Because Gitlab doesn’t delete created namespaces, we’ve done that manually. And after that, the job has failed, when trying to use one of the ‘stages’, for which the namespace was deleted.
Steps to replicate:
- Run the pipeline
- Delete the namespace created by the pipeline
- Run the pipeline again with the same parameter
Errors stating that you are unauthorized. The namespace is not created.
The namespace is created. No errors.