Gitlab shared agent - multi k8s cluster - not found context

Hi all,
I have Gitlab CE 14.10.5 self managed .
I have 2 k8s clusters , for ex: k8sclusterdev & k8sclusterprod
I have 2 project A1 & A2 in group A , each project has 2 branches , for ex : dev & prod
In project A1 , branch dev , I have file .gitlab/agents/k8sclusterdev-agent/config.yaml as below

ci_access:
  groups:
    - id: groupA

I registered agent successfully.
In project A1 & A2 , branch dev , I have .gitlab-ci.yaml job deploy_k8s as below

  script:
    - kubectl config get-contexts
    - kubectl config use-context groupA/projectA1:k8sclusterdev-agent

They works fine, cicd job at project A1 & A2 branch dev can find agent , get kubectl context.

I want to register another agent so project A1 & A2 branch prod can be deploy on k8sclusterprod.
In project A1 & A2 , branch prod , I have .gitlab-ci.yaml job deploy_k8s as below

  script:
    - kubectl config get-contexts
    - kubectl config use-context groupA/projectA1:k8sclusterprod-agent

I know that I have to create a new file .gitlab/agents/k8sclusterprod-agent/config.yaml and register it but which project should I put this file ?
I tried put it in project A1 or A2 branch prod, registered it successfully but online project A1 or A2 can use it , the remain one cannot find it, agent is not shared between projects in same group
Project that has it will find it like below

$ kubectl config get-contexts
CURRENT   NAME                                       CLUSTER   AUTHINFO   NAMESPACE
groupA/projectA1:k8sclusterdev-agent   gitlab    agent:51
groupA/projectA2:k8sclusterprod-agent   gitlab    agent:60
$ kubectl config use-context groupA/projectA2:k8sclusterprod-agent
Switched to context groupA/projectA2:k8sclusterprod-agent

The remain one cannot find it

$ kubectl config get-contexts
CURRENT   NAME   CLUSTER   AUTHINFO   NAMESPACE
$ kubectl config use-context groupA/projectA2:k8sclusterprod-agent
error: no context exists with the name: groupA/projectA2:k8sclusterprod-agent

Please give me some advice , thank you very much.

The situation you’re facing is due to the way GitLab Kubernetes agent registration and configuration are scoped within your GitLab setup. When you register a Kubernetes agent in GitLab, it’s typically associated with a specific project. This means the agent’s access and configurations are scoped to that project unless explicitly configured to be accessible by other projects within the same group or across groups.

In your case, you want two projects (A1 and A2) within the same group (Group A) to access two different Kubernetes clusters (k8sclusterdev and k8sclusterprod) based on their branch (dev or prod). Since the agent you’ve registered in one project isn’t automatically available in another, even within the same group, you’re encountering the issue where one project can access the agent, but the other cannot.

To resolve this and enable both projects to deploy to both Kubernetes clusters based on the branch, consider the following approach:

Centralise Kubernetes Agent Configuration
Since GitLab version 14.5, GitLab introduced the ability to define Kubernetes agent configurations at the group level, which would be the ideal solution for your case. However, if your GitLab CE version or your setup does not support or fully utilise group-level Kubernetes configurations, you might need to apply a workaround.

Workaround: Duplicate Agent Registration
As a workaround, if group-level agent configuration is not available or not working as expected, you can register the k8sclusterprod-agent in both projects (A1 and A2), but this approach has its downsides, such as the need to manage multiple instances of what is essentially the same agent configuration.

Register k8sclusterprod-agent for Both Projects: You would go to each project’s settings in GitLab and follow the process to register the Kubernetes agent for k8sclusterprod. This means you’ll have .gitlab/agents/k8sclusterprod-agent/config.yaml in both A1 and A2, tailored for each project.

Configure CI/CD for Branch-Specific Deployments: Ensure that your .gitlab-ci.yml in both projects correctly references the agent context for deployments. Since you’ve set this up already for one environment, it’s a matter of ensuring both projects are correctly set up to use the right agent context for the dev and prod branches.

Long-Term Solution: Group-Level Kubernetes Configuration
For a more scalable and cleaner solution, you should explore moving the Kubernetes agent configuration to the group level, if your GitLab version supports it. This would allow all projects within the group to share the same Kubernetes agent configuration, simplifying management and avoiding duplication. You might need to upgrade your GitLab CE version to access more advanced features like group-level Kubernetes configurations.

For immediate resolution, register the k8sclusterprod-agent with both projects and ensure your CI/CD configurations are correct.
Investigate upgrading GitLab or utilising group-level Kubernetes agent configurations for a more scalable solution.

Hi @doctor3182 , thank you for your reply
I think my Gitlab supported “Centralise Kubernetes Agent Configuration” or “Group-Level Kubernetes Configuration” because I’m using Gitlab CE 14.10.5 self managed ??
In fact, I have 10 projects from A1 to A10 in group A , in project A1 , branch dev , I have file .gitlab/agents/k8sclusterdev-agent/config.yaml

ci_access:
  groups:
    - id: groupA

So project A1 is picked to hold k8sclusterdev-agent for whole groupA
From project A1 → A10 , branch dev , I have .gitlab-ci.yaml job deploy_k8s as below

  script:
    - kubectl config get-contexts
    - kubectl config use-context groupA/projectA1:k8sclusterdev-agent

And 10 cicd jobs work fine, they can find kubectl context groupA/projectA1:k8sclusterdev-agent

However , when I pick project A2 to hold k8sclusterprod-agent for whole groupA , in project A2 , branch prod , I have file .gitlab/agents/k8sclusterprod-agent/config.yaml

ci_access:
  groups:
    - id: groupA

It doesn’t work as I expect , only project A2 cicd job can find kubectl context groupA/projectA2:k8sclusterprod-agent , 9 remain projects cannot.

For now, I apply a workaround as you said, I register 10 k8sclusterprod-agent (1 each project - branch prod)
for ex:

project A1
.gitlab/agents/k8sclusterprod-A1-agent/config.yaml

  script:
    - kubectl config get-contexts
    - kubectl config use-context groupA/projectA1:k8sclusterprod-A1-agent

project A2
.gitlab/agents/k8sclusterprod-A2-agent/config.yaml

  script:
    - kubectl config get-contexts
    - kubectl config use-context groupA/projectA2:k8sclusterprod-A2-agent

and they worked for now, but it’s inconvenient.

Please give me more hint, docs about “Centralise Kubernetes Agent Configuration” or “Group-Level Kubernetes Configuration” , I’m not sure what I’m doing is what you’re talking about.

Is it true that in a group there is only 1 k8s agent shared for projects in the group ?