i have a managed gitlab runner in AWS and everything was working today (verified with new checkins/builds). I created a new project and autodevops failed and i tracked it down to my Kubernetes environment was out of cpus so it couldnt deploy new app. I then noticed it had 100s of pods and namespaces and services of failed builds for the review/* job (different problem not looking for answer to that here) so i then proceeded to try and clean them up. My first go was to go into GitLab ui/environments and manually “stop” each review/* environment and then deleted each environment. I did this for roughly 200 review/* environments but this didnt clear up the pods running in Kubernetes so i then started to delete all the pods, stateful sets, service accounts, and namespaces. Its very probable i deleted the namespace this guy wants ‘myproject-6-review-sandbox-2jvyby’ but anyway, my current problem is that i can NOT get autodevops review builds to launch… it exceptions out before it gets to the gitlab runner and only log i found on the server is in the log file below
exceptions_json.log:
{“severity”:“ERROR”,“time”:“2020-07-13T23:27:39.214Z”,
“class”:“Ci::BuildPrepareWorker”,
“args”:[“7823”],
“retry”:3,
“queue”:“pipeline_processing:ci_build_prepare”,
“exception.class”:“ActiveRecord::RecordInvalid”,
“exception.message”:“Validation failed: Namespace has already been taken”,
“exception.backtrace”:[“app/services/clusters/kubernetes/create_or_update_namespace_service.rb:16:in execute'", "lib/gitlab/ci/build/prerequisite/kubernetes_namespace.rb:53:in create_namespace’”,
“lib/gitlab/ci/build/prerequisite/kubernetes_namespace.rb:17:in complete!'", "app/services/ci/prepare_build_service.rb:12:in each’”,
“app/services/ci/prepare_build_service.rb:12:in execute'", "app/workers/ci/build_prepare_worker.rb:13:in block in perform’”,
“app/workers/ci/build_prepare_worker.rb:12:in perform'", "lib/gitlab/with_request_store.rb:17:in enabling_request_store’”,
“lib/gitlab/with_request_store.rb:10:in with_request_store'", "lib/gitlab/sidekiq_daemon/monitor.rb:49:in within_job’”]}
The user interface shows
This job failed because the necessary resources were not successfully created. [ More information ]
I have tried again to delete the namespace i think it wants but that didn’t seem to resolve the issue. How do i correct my mistakes without burning everything to the ground (i.e. reinstall of GitLab and EKS)
tl;dr; i managed to modify the Kubernetes environment out from under GitLab and now they are out of sync