KUBECONFIG setup done ,but kubectl get pods -o wide not working from remote docker runner in gitlab

Infrastructure as Code & Cloud Native
1

Hi Gurus,

This is a bit tricky,I have a K8s cluster up and running and i am able to execute a docker image inside that cluster and i can see contents of command “kubectl get pods -o wide”(Note This cluster is up and running using AWS and i am able to see pods using kubectl get pods command ) .Now i have Gitlab setted up with this K8 cluster

I have set up variables $KUBE_URL $KUBE_USER and $KUBE_PASSWORD respectively in Gitlab with above K8 cluster

Here Gitlab runner console displays all these information as shown in console log below,at the end it fails for

$ kubeconfig=cluster1-config kubectl get pods -o wide

error: the server doesn’t have a resource type “pods”

ERROR: Job failed: exit code 1

Here is full console log

======================================================================================================

Running with gitlab-runner 11.4.2 (cf91d5e1)

on WotC-Docker-ip-10-102-0-70 d457d50a

Using Docker executor with image docker:latest …

Pulling docker image docker:latest …

Using docker image sha256:062267097b77e3ecf374b437e93fefe2bbb2897da989f930e4750752ddfc822a for docker:latest …

Running on runner-d457d50a-project-185-concurrent-0 via ip-10-102-0-70…

Fetching changes…

Removing cluster1-config

HEAD is now at 25846c4 Initial commit

From https://git.com/core-systems/gatling

25846c4…bcaa89b master → origin/master

Checking out bcaa89bf as master…

Skipping Git submodules setup

$ uname -a

Linux runner-d457d50a-project-185-concurrent-0 4.14.67-66.56.amzn1.x86_64 #1 SMP Tue Sep 4 22:03:21 UTC 2018 x86_64 Linux

$ apk add --no-cache curl

fetch http://dl-cdn.alpinelinux.org/alpine/v3.8/main/x86_64/APKINDEX.tar.gz

fetch http://dl-cdn.alpinelinux.org/alpine/v3.8/community/x86_64/APKINDEX.tar.gz

(1/4) Installing nghttp2-libs (1.32.0-r0)

(2/4) Installing libssh2 (1.8.0-r3)

(3/4) Installing libcurl (7.61.1-r1)

(4/4) Installing curl (7.61.1-r1)

Executing busybox-1.28.4-r1.trigger

OK: 6 MiB in 18 packages

$ curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl

% Total % Received % Xferd Average Speed Time Time Time Current

                             Dload  Upload   Total   Spent    Left  Speed

0 0 0 0 0 0 0 0 --:–:-- --:–:-- --:–:-- 0

95 37.3M 95 35.8M 0 0 37.8M 0 --:–:-- --:–:-- --:–:-- 37.7M

100 37.3M 100 37.3M 0 0 38.3M 0 --:–:-- --:–:-- --:–:-- 38.3M

$ chmod +x ./kubectl

$ mv ./kubectl /usr/local/bin/kubectl

$ kubectl config set-cluster nosebit --server="$KUBE_URL" --insecure-skip-tls-verify=true

Cluster “nosebit” set.

$ kubectl config set-credentials admin --username="$KUBE_USER" --password="$KUBE_PASSWORD"

User “admin” set.

$ kubectl config set-context default --cluster=nosebit --user=admin

Context “default” created.

$ kubectl config use-context default

Switched to context “default”.

$ cat $HOME/.kube/config

apiVersion: v1

clusters:

contexts:

  • context:

    cluster: nosebit

    user: admin

    name: default

current-context: default

kind: Config

preferences: {}

users:

  • name: admin

    user:

    password: |-

    MIIDOzCCAiOgAwIBAgIJALOrUrxmhgpHMA0GCSqGSIb3DQEBCwUAMBgxFjAUBgNV

BAMMDTEzLjU4LjE3OC4yNDEwHhcNMTgxMTI1MjIwNzE1WhcNMjgxMTIyMjIwNzE1

WjAYMRYwFAYDVQQDDA0xMy41OC4xNzguMjQxMIIBIjANBgkqhkiG9w0BAQEFAAOC

AQ8AMIIBCgKCAQEA4jmyesjEiy6T2meCdnzzLfSE1VtbY//0MprL9Iwsksa4xssf

PXrwq97I/aNNE2hWZhZkpPd0We/hNKh2rxwNjgozQTNcXqjC01ZVjfvpvwHzYDqj

4cz6y469rbuKqmXHKsy/1docA0IdyRKS1JKWz9Iy9Wi2knjZor6/kgvzGKdH96sl

ltwG7hNnIOrfNQ6Bzg1H6LEmFP+HyZoylWRsscAIxD8I/cmSz7YGM1L1HWqvUkRw

GE23TXSG4uNYDkFaqX46r4nwLlQp8p7heHeCV/mGPLd0QCUaCewqSR+gFkQz4nYX

l6BA3M0Bo4GHMIGEMB0GA1UdDgQW

BBQqsD7FUt9vBW2LcX4xbqhcO1khuTBIBgNVHSMEQTA/gBQqsD7FUt9vBW2LcX4x

bqhcO1khuaEcpBowGDEWMBQGA1UEAwwNMTMuNTguMTc4LjI0MYIJALOrUrxmhgpH

MAwGA1UdEwQFMAMBAf8wCwYDVR0PBAQDAgEGMA0GCSqGSIb3DQEBCwUAA4IBAQAY

6mxGeQ90mXYdbLtoVxOUSvqk9+Ded1IzuoQMr0joxkDz/95HCddyTgW0gMaYsv2J

IZVH7JQ6NkveTyd42QI29fFEkGfPaPuLZKn5Chr9QgXJ73aYrdFgluSgkqukg4rj

rrb+V++hE9uOBtDzcssd2g+j9oNA5j3VRKa97vi3o0eq6vs++ok0l1VD4wyx7m+l

seFx50RGXoDjIGh73Gh9Rs7/Pvc1Pj8uAGvj8B7ZpAMPEWYmkkc4F5Y/14YbtfGc

2VlUJcs5p7CbzsqI5Tqm+S9LzZXtD1dVnsbbbGqWo32CIm36Cxz/O/FCf8tbITpr

u2O7VjBs5Xfm3tiW811k

    username: eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6ImRlZmF1bHQtdG9rZW4tdzZqdDYiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiZGVmYXVsdCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjFiMjc2YzIxLWYxMDAtMTFlOC04YjM3LTAyZDhiMzdkOTVhMCIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNQifQ.RCQQWjDCSkH8YckBeck-EIdvOnTKBmUACXVixPfUp9gAmUnit5qIPvvFnav-C-orfYt552NQ5GTLOA3yR5-jmxoYJwCJBfvPRb1GqqgiiJE2pBsu5Arm30MOi2wbt5uCNfKMAqcWiyJQF98M2PFc__jH6C1QWPXgJokyk7i8O6s3TD69KrrXNj_W4reDXourLl7HwHWoWwNKF0dgldanug-_zjvE06b6VZBI-YWpm9bpe_ArIOrMEjl0JRGerWahcQFVJsmhc4vgw-9-jUsfKPUYEfDItJdQKyV9dgdwShgzMINuuHlU7w7WBxmJT6cqMIvHRnDHuno3qMKTJTuh-g

$ kubectl config view --minify > cluster1-config

$ export KUBECONFIG=$HOME/.kube/config

$ kubectl --kubeconfig=cluster1-config config get-contexts

CURRENT NAME CLUSTER AUTHINFO NAMESPACE

  •     default   nosebit   admin

$ kubeconfig=cluster1-config kubectl get pods -o wide

error: the server doesn’t have a resource type “pods”

ERROR: Job failed: exit code 1

==================================================================================================

Here is my .gitlab-ci.yml content,could you suggest why kubectl get pods not displaying pods of remote cluster even when KUBECONFIG set up is done successfully using aws

image : docker:latest

variables:

CONTAINER_DEV_IMAGE: Docker Hub

stages:

  • deploy

deploy:

stage: deploy

tags:

- docker

script:

  • ‘uname -a’

  • ‘apk add --no-cache curl’

  • ‘curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl’

  • ‘chmod +x ./kubectl’

  • ‘mv ./kubectl /usr/local/bin/kubectl’

  • ‘kubectl config set-cluster nosebit --server="$KUBE_URL" --insecure-skip-tls-verify=true’

  • ‘kubectl config set-credentials admin --username=" " --password="$KUBE_PASSWORD"’

  • ‘kubectl config set-context default --cluster=nosebit --user=admin’

  • ‘kubectl config use-context default’

  • ‘cat $HOME/.kube/config’

  • ‘kubectl config view --minify > cluster1-config’

  • ‘export KUBECONFIG=$HOME/.kube/config’

  • ‘kubectl --kubeconfig=cluster1-config config get-contexts’

  • 'kubeconfig=cluster1-config kubectl get pods -o wide ’

Why gitlab runner failing to get pods from Kubernetes cluster(Note This cluster is up and running using AWS and i am able to see pods using kubectl get pods command )