Gtlab-ci running kubectl exec hangs *
Describe your question in as much detail as possible:
We are running django test from kubernetes pod using:
/usr/local/bin/kubectl exec -it deployment/${POD}-n ${NAMESPACE} --request-timeout=3h --kubeconfig ${CONFIG_FILE_PATH}-- /usr/bin/python3.7 manage.py test vulcan --verbosity 3
The command executed as expected and run, but after a while, the django finish to run his “Running migrations” step, the django “test” step is not starting and the gitlab-ci hangs, and continue to the next command in the pipeline. (the tests command that run on the kubernetes pod still running). just the output hangs…
What am i seeing?
But when i am running the same command mention above in the runner itself using terminal i can see all the outputs.
In this screenshot you can see that the django migration step finished but then there is no running test, just the next command in the pipeline:
What is the expected behavior?
- That gitlab-ci will not hang up until the command will finished.
- The full output of the command.
In this screenshot you can see the command output from the runner terminal (this is good output):
So, because the same command runs as expected in the runner terminal, i believe there is problem in gitlab-ci.
The official answer i got from the gitlab support was that “It seems that kubectl exec is known to randomly stop streaming data”.
sorry for the language, but this is bulshit! because kubectl not randomly hangs, and if it does, then it should hangs also when running it from the runner terminal.
looks like the gitlab support guy just wanted to go home erlier and not proffesional.
- Kubernetes version (kops): 1.18
- Gitlab Runner version: 13.5.0