Confusion about roles of Gitlab k8s agent and Flux for the manifests deployment

(not sure my question belongs here)

Hi there and happy new year

I’ve read page GitOps with GitLab: What you need to know about the Flux CD integration page Tutorial: Set up Flux for GitOps | GitLab, page Using GitOps with a Kubernetes cluster | GitLab and I viewed the 2 youtube videos that explain how to setup flux on gitlab but I’m failing to understand what brings adding the agentk to a kubernetes cluster. The video https://www.youtube.com/watch?v=EjPVRM-N_PQ does not mention the installation of the gitlab kubernetes agent.
I’m also confused because in the past (about 1 year ago), I’ve tested gitlab agent for kubernetes to deploy manifests in a namespace, it was almost doing the same job.

Even after having played a couple with flux in gitlab, I don’t understand the benefits.

Can someone clarify all what the role of these components ? Iis gitlab k8s agent still necessary ?

Best

It’s a relief ! I am not the only one confused !

I also find this turn very ambiguous actually.
Apparently, @vnagy works a lot on GitOps with Gitlab. Maybe he can help !
Following is my understanding of the vision. Please somebody correct me if I am wrong :

1. Era of Agentk / Gitlab 13.4 - 13.10 (Using GitOps with the agent for Kubernetes (deprecated) | GitLab)

Enables connection between Gitlab and Kubernetes.
GitOps in a “push” method : Gitlab (formerly agentk) watchs Git repositories and ask for changes in Kubernetes. Like in this diagram : Using GitOps with the agent for Kubernetes (deprecated) | GitLab

2. Use Agentk in pipelines / Gitlab 14.1 - 14.2 (Using GitLab CI/CD with a Kubernetes cluster | GitLab)

Facilitates connections to Kubernetes cluster from pipelines (run kubectl commands through the agent without managing authentication).
Enables to push manifests via CI/CD

3. Arrival of Flux / Gitlab 16.1 (Using GitOps with a Kubernetes cluster | GitLab)

GitOps in a “pull” method : Flux watchs Git repositories and apply manifests.
This implies that communication is reversed : Kubernetes (formerly Flux) connects to Gitlab to get source code. No more need for Gitlab to connect to Kubernetes

Now, new problem : Switching to asynchronous deployment : the problem with this approach is that it breaks synchronicity in pipelines : you declare changes in manifests, then you have to wait for Flux to apply, and you cannot procede following tasks (like acceptance tests for example). There is an issue for that : CI support for triggering a Flux sync (#405007) · Issues · GitLab.org / GitLab · GitLab

Gitlab seems to recommend using Flux and agentk together in this diagram Using GitOps with a Kubernetes cluster | GitLab, which likely seems to resolve this synchronicity problem with notifications. Is it really the intention ?

Gitlab also claims in this same page :

You should use both Flux and agentk for GitOps deployments. Flux keeps the cluster state synchronized with the source, while agentk simplifies the Flux setup, provides cluster-to-GitLab access management, and visualizes the cluster state in the GitLab UI.

And now we arrive to a new feature provided by agentk :

4. Dashboard for Kubernetes / Gitlab 16.1 (Dashboard for Kubernetes | GitLab)

This aims to give a UI to observe reconcilation states of Flux resources.
It uses agentk to ask the cluster about the resources.

So agentk seems to have changed from a GitOps controller to a “connection agent” providing observability on Kubernetes.

While it is not mandatory to keep agentk once Flux is used, it may allow new services (synchronicity ?, observability).

But I would really appreciate Gitlab staff point of view on this also !

So just to see that I understood this correctly;

I need to run flux bootstrap for every project/repo that should deploy something into k8s?

That sounds extremely tedious. I was hoping the agent would take that part off my hands and not just provide “dashboards”.

Or am I missing something again?

It’s really unclear/confusing how flux/agent work together and your post has been great clearing some confusion but I’m still trying to wrap my head around why I even need the agent when I have to setup flux (and/or runners) anyway

//EDIT:

So looking at the AWS HandsOn Lab it shows a bit more what all the “hidden” requirements are.

Basically you need GitLab Runner setup somewhere, then you need flux setup (per deployable repo with respective manifests), then you need app repo with code and build-instructions that pushes the image into deployable-repo which then triggers flux.

I honestly do not understand why this should be done that way. Having a hard requirement on GitLab Runner already, I might just deploy that runner into the k8s env and skip flux+agent altogether. Whether flux calls kustomize/helm or the runner calls it doesnt matter to me and it cuts down the complexity of the setup manyfold.

Nope. flux bootstrap push the Flux manifests to a Git repository and deploy Flux on the cluster.
So the intent is to deploy a GitOps controller in K8S. You may want to declare only once per K8S.

IMHO, agent purpose, before Flux arrival, was to sync yaml in git repo with K8S deployments.
Now there is Flux, you do not need agent anymore for deployments (I am actually declaring deployments synced with Flux without any agent).
But Gitlab seem to promote the agent for other purposes now : I can see immediate reconcilation (just a way to speed up synchronization) and Dashboard for Kubernetes (to provide a UI integrated into Gitlab of what is actually deployed/reconciled/unsync, since Flux do not provide an official UI)

Well thank you. But I have to confess it is still confusing for me too. And my clarification is only a proposition. I am not a Gitlab staff member. That’s why I asked for a confirmation to @vnagy

Gitlab Runner is out of scope. It is just an executor of a CI/CD pipeline job. You need it only if you build something (docker image, helm chart, OCI artefact, etc.).
You can deploy things with Flux without any runner.

Nope. You need Flux setup once.
Then in your Flux setup Git repository, you can declare other Git Repositories containing manifests.
Maybe you can setup Flux more than once but it would lead to several GitOps controllers in your K8S. It could be a use case in certain circumstances.

You effectively need app repo if you develop your app.
But the built artefacts do not necessarily trigger Flux : it depends what you are tracking with Flux as a Source.
If your Source is an OCI artefact, then yes. If your Source is a Git Repository, then you need to update your manifests to trigger Flux.

News from Gitlab 17 (A guide to the high-impact breaking changes in GitLab 17.0) :

• The pull-based deployment features of the GitLab agent for Kubernetes is deprecated
  • Impacts projects using the GitLab agent for Kubernetes for deployments.
  • The change may break CD workflows relying on the GitLab agent for Kubernetes.
  • The agent itself is not deprecated and still used for a number of features, like communicating with the cluster, its API endpoints and pushing information about events in the cluster to GitLab.

As I thought, Agent is not used anymore as a GitOps Controller. But it is still used to interconnect Gitlab and Kubernetes (See Dashboard for Kubernetes)