Gitlab + K3s + Traefik: Cannot get resource "ingressroutes" in API group "traefik.containo.us" in the namespace "traefik"

:thinking: Issue description

I am getting the error below when I attempt to have my alpha.test container deployed via Gitlab on my K3s cluster.

Error I’m getting:
This appears in the Job logs:

Error from server (Forbidden): error when retrieving current configuration of:
Resource: "traefik.containo.us/v1alpha1, Resource=ingressroutes", GroupVersionKind: "traefik.containo.us/v1alpha1, Kind=IngressRoute"
Name: "alpha-whoami", Namespace: "traefik"
from server for: "deployment.yml": ingressroutes.traefik.containo.us "alpha-whoami" is forbidden: User "system:serviceaccount:alpha-test-8-production:alpha-test-8-production-service-account" cannot get resource "ingressroutes" in API group "traefik.containo.us" in the namespace "traefik"

:earth_americas: Environment

Version info

  • Gitlab Version: 13.8.3 (self-managed)
  • Gitlab Runner: 13.8.0
  • K3s Version: 1.19.7

:man_dancing: Steps to reproduce the issue

  1. Provision brand new cluster
  2. Connect Gitlab to cluster
  3. Run deployment

:pray: What is expected?

I would like my alpha.test container to deploy in its own namespace, but accept web traffic via the Ingress Controller in the traefik namespace.

:triumph: Steps I have taken already to try and fix

  • Hardcode the namespace into the default namespace (Gitlab kept wanting to do its own project namespace)
  • Delete the service account and re-add it
  • Delete the cluster and re-add it (following these docs)

:sparkles: Additional details

Gitlab service account

I made sure I followed these steps and created the Gitlab service account with this:
Gitlab Service Account settings:

gitlab-admin-service-account.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: gitlab
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: gitlab-admin
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
  - kind: ServiceAccount
    name: gitlab
    namespace: kube-system

How I installed Traefik:

I use Ansible to deploy Traefik via Helm:

ansible-playbook.yaml
---
- name: Add Traefik's chart repository.
  community.kubernetes.helm_repository:
    name: traefik
    repo_url: "https://helm.traefik.io/traefik"

- name: Install Traefik with Helm.
  community.kubernetes.helm:
    name: traefik
    chart_ref: traefik/traefik
    update_repo_cache: yes
    release_namespace: traefik
    create_namespace: yes
    values: "{{ lookup('template', 'traefik-chart-values.yaml') | from_yaml }}"

It calls this values file:

traefik-chart-values.yaml
#
# Advanced K8s settings
#
rbac:
  enabled: true
tolerations:
  - key: "CriticalAddonsOnly"
    operator: "Exists"
  - key: "node-role.kubernetes.io/master"
    operator: "Exists"
    effect: "NoSchedule"

#
# Configure providers
#
providers:
  kubernetesIngress:
    publishedService:
      enabled: true
#
# Create an IngressRoute for the dashboard
#
ingressRoute:
  dashboard:
    enabled: true

# Configure ports
ports:
  # The name of this one can't be changed as it is used for the readiness and
  # liveness probes, but you can adjust its config to your liking
  traefik:
    port: 9000
    # Use hostPort if set.
    # hostPort: 9000
    #
    # Use hostIP if set. If not set, Kubernetes will default to 0.0.0.0, which
    # means it's listening on all your interfaces and all your IPs. You may want
    # to set this value if you need traefik to listen on specific interface
    # only.
    # hostIP: 192.168.100.10

    # Defines whether the port is exposed if service.type is LoadBalancer or
    # NodePort.
    #
    # You SHOULD NOT expose the traefik port on production deployments.
    # If you want to access it from outside of your cluster,
    # use `kubectl port-forward` or create a secure ingress
    expose: true
    # The exposed port for this service
    exposedPort: 9000
    # The port protocol (TCP/UDP)
    protocol: TCP
  web:
    port: 8000
    # hostPort: 8000
    expose: true
    exposedPort: 80
    # The port protocol (TCP/UDP)
    protocol: TCP
    # Use nodeport if set. This is useful if you have configured Traefik in a
    # LoadBalancer
    # nodePort: 32080
    # Port Redirections
    # Added in 2.2, you can make permanent redirects via entrypoints.
    # https://docs.traefik.io/routing/entrypoints/#redirection
    # redirectTo: websecure
  websecure:
    port: 8443
    # hostPort: 8443
    expose: true
    exposedPort: 443
    # The port protocol (TCP/UDP)
    protocol: TCP
    # nodePort: 32443
    # Set TLS at the entrypoint
    # https://doc.traefik.io/traefik/routing/entrypoints/#tls
    tls:
      enabled: true
      # this is the name of a TLSOption definition
      options: ""
      certResolver: ""
      domains: []
      # - main: example.com
      #   sans:
      #     - foo.example.com
      #     - bar.example.com

#
# Things that I felt more comfortable setting via the CLI
#
additionalArguments:
  - "--certificatesresolvers.letsencrypt.acme.email=myemail@email.test"
  - "--certificatesresolvers.letsencrypt.acme.storage=/data/acme.json"
  - "--certificatesresolvers.letsencrypt.acme.caserver=https://acme-staging-v02.api.letsencrypt.org/directory"
  - "--api.insecure=true"
  - "--accesslog=true"
  - "--log.level=ERROR"

My deployment

.gitlab-ci.yml
services:
  - name: docker:dind
    alias: docker

stages:
  - deploy

deploy to production:
  stage: deploy
  image: 
    name: bitnami/kubectl:latest
    entrypoint: [""]
  script:
    - kubectl apply -f deployment.yml
  environment:
    name: production
deployment.yaml
---
apiVersion: v1
kind: Pod
metadata:
  name: alpha-whoami
  labels:
    app: alpha-whoami
    namespace: alpha-test-8-production
spec:
  containers:
    - name: alpha-whoami
      image: containous/whoami:latest
      ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: alpha-whoami
  namespace: alpha-test-8-production
spec:
  ports:
    - port: 80
      protocol: TCP
      targetPort: 80
  selector:
    app: alpha-whoami
  type: ClusterIP
---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: alpha-whoami
  namespace: traefik
spec:
  entryPoints:
    - web
  routes:
    - match: Host(`alpha.test`)
      kind: Rule
      services:
        - name: alpha-whoami
          port: 80

Thank you for your help!

If there is anything else that you would like to see, just let me know! Thanks!!! :raised_hands:

Hi, i think i run in the same error and i find a way to resolve it.
As far as i understood, Kubernetes will create a namespace for your deployment and a service account. But this service account has limited access and can’t access all ressources on kubernetes.

I have set my project as “Cluster management project (alpha)” and with this, i get cluster wide permissions. Go to you Kubernetes Cluster on GitHub and then to “Advanced Settings”. Here you can set your project as “Cluster management project”.
This is more or less a workaround, the correct solution should be the correct permissions for the service account.

Thanks @benny! This is a huge help. I made the same post on Traefik’s forums and between your posts and jakubhajek, I think we’re onto something.

I think next step is figuring out how to get this role bound to the service account that is automatically created by Gitlab.

If you know of anything, feel free to post more thoughts. Otherwise I will keep playing around with this and will report back with my findings. Thanks! :raised_hands:

I believe this has something to do with the service accounts created by Gitlab do not have permissions to access IngressRoutes.

I created a new post asking if there is a way to do that. If I find a solution, I will post back here what worked with me specifically for Traefik.

I’m happy to report, I found my solution.

Long story short:

  • Use Ingress, not IngressRoute

More detail:

After days of going back and forth, it all came down to changing my deployment to use Ingress instead of IngressRoute.

My old deployment file had:

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: alpha-whoami
  namespace: traefik
spec:
  entryPoints:
    - web
  routes:
    - match: Host(`alpha.test`)
      kind: Rule
      services:
        - name: alpha-whoami
          port: 80

I changed this to be:

kind: Ingress
apiVersion: extensions/v1beta1
metadata:
  name: ingress-alpha
spec:
  rules:
    - host: alpha.test
      http:
        paths:
          - path: /
            backend:
              serviceName: alpha-whoami
              servicePort: 80

For total clarity, here is my full deployment file that is working:

working-deployment.yml
---
apiVersion: v1
kind: Pod
metadata:
  name: alpha-whoami
  labels:
    app: alpha-whoami
spec:
  containers:
    - name: alpha-whoami
      image: containous/whoami:latest
      ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: alpha-whoami
spec:
  ports:
    - port: 80
      protocol: TCP
      targetPort: 80
  selector:
    app: alpha-whoami
  type: ClusterIP
---
kind: Ingress
apiVersion: extensions/v1beta1
metadata:
  name: ingress-alpha
spec:
  rules:
    - host: alpha.test
      http:
        paths:
          - path: /
            backend:
              serviceName: alpha-whoami
              servicePort: 80

Important notes

Gut feeling why this happened

  • Gitlab is increasing the security of the runners by limiting access to certain Kubernetes functions
  • Runners have limited permissions (they do not run as cluster-admin anymore)
  • Traefik is using an IngressRoute CRD (not exactly sure why?)
  • Gitlab is expecting Ingress when they create service accounts and does not have permission to IngressRoute

Using Ingress fixed my issue. I am now able to run automated deployments between Gitlab and my K8s cluster with Traefik :partying_face:

Hope this helps someone else!