Gitlab agent unable to run kubectl commands through ci cd tunnel

From CI / CD pipeline, unable to execute kubectl commands (get pods, namespaces, etc…)

I wanted to deploy the code to staging environment and configured the gitlab-ci.yaml accordingly. I have configured and registered the agent (.gitlab.agents..config.yaml) and installed gitlab-agent in my kubernetes cluster using grpc url of KAS (grpc://:port) successfully.

My gitlab-ci.yml looks like below:

stages:
  - admin_staging

.kube-context:
  before_script:
    - if [-n "$KUBE_CONTEXT" ]; then kubectl config use-config "$KUBE_CONTEXT"; fi

Deploy admin staging:
  stage: admin_staging
  extends: [.kube-context]
  variables:
    KUBE_CONTEXT: "<project>:<admin-agent>"
    KUBE_NAMESPACE: admin-cluster
  image:
    name: bitnami/kubectl:latest
    entrypoint: [""]
  before_script:
    - kubectl config use-context $KUBE_CONTEXT
    - kubectl config get-contexts
    - kubectl config view
    - kubectl get ns
    - kubectl get pods --all-namespaces

I was able to change context and view config but for other commands like get ns and get pods I am getting below error message.

E0228 07:46:36.475496      33 memcache.go:238] couldn't get current server API group list: the server responded with the status code 426 but did not return more information

Screenshot from pipeline:

image1246×644 75.4 KB

Connection to agent:

image1284×369 27.8 KB

• What version are you on? Are you using self-managed or GitLab.com?

  • GitLab (Hint: /help): 15.8 self-managed

• What troubleshooting steps have you already taken? Can you link to any docs or other resources so we know where you have been?
  Not sure if it is related to how KAS is connected to agent so I tried changing KAS connection from grpc to ws/wss but it is not working giving all sorts of connection issues.

Please help if anybody is aware of this issue and its resolution. Thanks.

After following the instructions from #245 , I was able to resolve nginx issue but now I am getting another error - "GitLab Agent Server: Unauthorized: Authorization header: expecting token". Detailed logs mentioned in the ticket.

Any suggestions on this would be appreciated.

I figured that my default config is trying to send the request to http instread of https, so I changed the config file as below and used it while executing the kubectl command.

.kubectl_config: &kubectl_config
 - |
   cat << EOF > "$KUBE_CFG_FILE"
   apiVersion: v1
   kind: Config
   clusters:
   - name: agent
     cluster:
       server: https://kas.local.io/k8s-proxy/
   users:
   - name: agent
     user:
       token: "ci:5:$CI_JOB_TOKEN"
       server: https://kas.local.io/k8s-proxy/
   users:
   - name: agent
     user:
       token: "ci:$AGENT_ID:$CI_JOB_TOKEN"
   contexts:
   - name: agent
     context:
       cluster: agent
       user: agent
   current-context: agent
   EOF

  before_script:
    - *kubectl_config
    - kubectl --kubeconfig="$KUBE_CFG_FILE" -v 10 get ns

Now I am getting error Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"GitLab Agent Server: Bad request: URL does not start with expected prefix","reason":"BadRequest","code":400}

Also added location configuration in nginx config file for k8s-proxy

location /k8s-proxy/ {
   proxy_pass http://<ip>:<port>/;
   proxy_http_version 1.1;
   proxy_set_header X-Real-IP $remote_addr;
   proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
   proxy_set_header X-NginX-Proxy true;
   proxy_set_header    X-Forwarded-Ssl on;
   proxy_set_header Host $host;
   proxy_set_header Upgrade "websocket";
   proxy_set_header Connection $connection_upgrade;
   #proxy_set_header Sec-WebSocket-Protocol $http_sec_websocket_protocol;
   #proxy_set_header Sec-WebSocket-Extensions $http_sec_websocket_extensions;
   #proxy_set_header Sec-WebSocket-Key $http_sec_websocket_key;
   #add_header Sec-WebSocket-Version "";

   access_log /var/log/nginx/kas.local.io.access.log;
   error_log /var/log/nginx/kas.local.io.error.log;
}

Still no luck.