GitLab runner cannot access k8s resources in deploy stage

Recently I installed gitlab runner in my k8s cluster and configured it with gitlab ci cd. The build stage that consists of docker login, build, and push commands is working. But the deploy stage that consists of deploying the k8s.yaml file is stuck in forbidden service account to cluster resources errors.

my gitlab-ci.yaml file:

# @format

image: docker:19.03.1

services:
  - docker:19.03.1-dind






variables:
  DOCKER_DRIVER: overlay2
  DOCKER_HOST: tcp://localhost:2375
  DOCKER_TLS_CERTDIR: ''
  GIT_SUBMODULE_STRATEGY: recursive

stages:
  - build
  - deploy

## Templates

.build:
  stage: build
  tags:
    - kubernetes
  script:
    - cd $SOURCE_PATH
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
    - docker build -t "$IMAGE" -t "$CI_REGISTRY_IMAGE/$ENV:latest" --build-arg NODE_ENV=${ENV} --network host .
    - docker push "$IMAGE"
    - docker push "$CI_REGISTRY_IMAGE/$ENV:latest"

.deploy:
  stage: deploy
  image: alpine
  tags:
    - kubernetes
  script:
    - cd $SOURCE_PATH
    - apk add --no-cache curl gettext
    - curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl
    - chmod +x ./kubectl
    - mv ./kubectl /usr/local/bin/kubectl
    - envsubst < $K8S_APP_CONFIG > ./k8s.yml
    - kubectl apply -f k8s.yml

.stage:
  variables:
    ENV: 'stage'
    REPLICAS: 1
    HOST_URL: <...>
  only:
    refs:
      - stage

.prod:
  variables:
    ENV: 'prod'
    REPLICAS: 1
    HOST_URL: <...>
  only:
    refs:
      - prod

.app:
  variables:
    APP_NAME: 'test-mig'
    K8S_APP_CONFIG: 'ci/k8s-services.yml'
    CI_REGISTRY_IMAGE: 'registry.gitlab.com/<...>'
    IMAGE: $CI_REGISTRY_IMAGE/$ENV:$CI_PIPELINE_IID
    SOURCE_PATH: '.'
    PORT: 3000
    NAMESPACE: $APP_NAME-$ENV

build-prod:
  extends:
    - .app
    - .prod
    - .build

deploy-prod:
  extends:
    - .app
    - .prod
    - .deploy
  when: manual
  needs:
    - build-prod

build-stage:
  extends:
    - .app
    - .stage
    - .build

deploy-stage:
  extends:
    - .app
    - .stage
    - .deploy
  needs:
    - build-stage

Role yaml file:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: gitlab-runner
  namespace: gitlab-runner
rules:
  - apiGroups: ["*"] 
    resources: ["*"]
    verbs: ["list", "get", "watch", "create", "delete", "patch", "update"]

rolebinding to default service account command:

kubectl create rolebinding --namespace=gitlab-runner gitlab-runner-binding --role=gitlab-runner --serviceaccount=gitlab-runner:default

Error in deploy stage:

$ kubectl apply -f k8s.yml
Error from server (Forbidden): error when retrieving current configuration of:
Resource: "/v1, Resource=namespaces", GroupVersionKind: "/v1, Kind=Namespace"
Name: "test-mig-stage", Namespace: ""
from server for: "k8s.yml": namespaces "test-mig-stage" is forbidden: User "system:serviceaccount:gitlab-runner:default" cannot get resource "namespaces" in API group "" in the namespace "test-mig-stage"
Error from server (Forbidden): error when retrieving current configuration of:
Resource: "/v1, Resource=secrets", GroupVersionKind: "/v1, Kind=Secret"
Name: "gitlab", Namespace: "test-mig-stage"
from server for: "k8s.yml": secrets "gitlab" is forbidden: User "system:serviceaccount:gitlab-runner:default" cannot get resource "secrets" in API group "" in the namespace "test-mig-stage"
Error from server (Forbidden): error when retrieving current configuration of:
Resource: "apps/v1, Resource=deployments", GroupVersionKind: "apps/v1, Kind=Deployment"
Name: "test-mig", Namespace: "test-mig-stage"
from server for: "k8s.yml": deployments.apps "test-mig" is forbidden: User "system:serviceaccount:gitlab-runner:default" cannot get resource "deployments" in API group "apps" in the namespace "test-mig-stage"
Error from server (Forbidden): error when retrieving current configuration of:
Resource: "/v1, Resource=services", GroupVersionKind: "/v1, Kind=Service"
Name: "test-mig-service", Namespace: "test-mig-stage"
from server for: "k8s.yml": services "test-mig-service" is forbidden: User "system:serviceaccount:gitlab-runner:default" cannot get resource "services" in API group "" in the namespace "test-mig-stage"
Error from server (Forbidden): error when retrieving current configuration of:
Resource: "extensions/v1beta1, Resource=ingresses", GroupVersionKind: "extensions/v1beta1, Kind=Ingress"
Name: "test-mig-ingress", Namespace: "test-mig-stage"
from server for: "k8s.yml": ingresses.extensions "test-mig-ingress" is forbidden: User "system:serviceaccount:gitlab-runner:default" cannot get resource "ingresses" in API group "extensions" in the namespace "test-mig-stage"
Running after_script
00:00
Uploading artifacts for failed job
00:00
ERROR: Job failed: command terminated with exit code 1

Hi @ali.j.alhajj

your Role grants access to GitLab Runner only within gitlab-runner namespace. You need to create proper Role and ClusterRole which gives access you need.

Hello @balonik , I really appreciate writing an example to me.

hey @balonik

I created this clusterrolebinding and the pipeline worked but not sure if it’s the best way. I made it work using this command:

kubectl create clusterrolebinding gitlab-runner-binding --clusterrole=cluster-admin --serviceaccount=gitlab-runner:default

Hi @ali.j.alhajj

Please look at official k8s docs regarding RBAC and use the best what suits your use-case. Using RBAC Authorization | Kubernetes

It’s impossible to suggest the best RBAC setup without knowing your environment.