Immediate reconciliation not working with gitlab-agent and fluxcd

General
1

Immediate reconciliation often works ok at the beginning but after random time quickly fails : I don’t see any fluxcd reconciliation and no events are generated in k8s.

After delete of gitlab-agent pod, it works again then fails again.

 

Around 2025-05-21T09:38 is the time there should have been an immediate reconciliation after new commit on gitops repo

{"time":"2025-05-21T09:37:43.769892776Z","level":"INFO","msg":"Module is disabled. Set the 'OCS_ENABLED' environment variable to 'true` to enable it","mod_name":"starboard_vulnerability"}
{"time":"2025-05-21T09:37:43.828965901Z","level":"INFO","msg":"Starting","mod_name":"agent_registrar"}
{"time":"2025-05-21T09:37:43.829457331Z","level":"INFO","msg":"Starting","mod_name":"google_profiler"}
{"time":"2025-05-21T09:37:43.82955713Z","level":"INFO","msg":"Starting","mod_name":"agentk2kas_tunnel"}
{"time":"2025-05-21T09:37:43.829574327Z","level":"INFO","msg":"Starting","mod_name":"gitops-manifest"}
{"time":"2025-05-21T09:37:43.829570918Z","level":"INFO","msg":"Starting","mod_name":"observability"}
{"time":"2025-05-21T09:37:43.830375117Z","level":"INFO","msg":"Observability endpoint is up","mod_name":"observability","net_network":"tcp","net_address":":8080"}
{"time":"2025-05-21T09:37:50.450635815Z","level":"INFO","msg":"attempting to acquire leader lease flux-system/agent-12-lock...","agent_id":12}
{"time":"2025-05-21T09:37:50.505234267Z","level":"INFO","msg":"successfully acquired lease flux-system/agent-12-lock","agent_id":12}
{"time":"2025-05-21T09:37:50.505805119Z","level":"INFO","msg":"Event occurred","agent_id":12,"object":{"name":"agent-12-lock","namespace":"flux-system"},"fieldPath":"","kind":"Lease","apiVersion":"coordination.k8s.io/v1","type":"Normal","reason":"LeaderElection","message":"gitlab-agent-v2-5d6d4867b-kmtg2 became leader"}
{"time":"2025-05-21T09:37:50.505908362Z","level":"INFO","msg":"Starting","mod_name":"remote_development","agent_id":12}
{"time":"2025-05-21T09:37:50.505925622Z","level":"INFO","msg":"Starting","mod_name":"flux","agent_id":12}
{"time":"2025-05-21T09:37:50.506339877Z","level":"INFO","msg":"Starting GitRepository controller","mod_name":"flux","agent_id":12}
{"time":"2025-05-21T09:38:50.526911688Z","level":"ERROR","msg":"Error handling a connection","mod_name":"agentk2kas_tunnel","agent_id":12,"error":"rpc error: code = Unavailable desc = error reading from server: failed to get reader: failed to read frame header: EOF"}
{"time":"2025-05-21T09:38:50.527032468Z","level":"ERROR","msg":"Error handling a connection","mod_name":"agentk2kas_tunnel","agent_id":12,"error":"rpc error: code = Unavailable desc = error reading from server: failed to get reader: failed to read frame header: EOF"}
{"time":"2025-05-21T09:38:50.527085416Z","level":"WARN","msg":"GetConfiguration.Recv failed","agent_id":12,"error":"rpc error: code = Unavailable desc = error reading from server: failed to get reader: failed to read frame header: EOF"}
{"time":"2025-05-21T09:38:50.527184194Z","level":"ERROR","msg":"Failed to receive project to reconcile","mod_name":"flux","agent_id":12,"error":"rpc error: code = Unavailable desc = error reading from server: failed to get reader: failed to read frame header: EOF"}

 

 

  • gitlab-agent chart :

 
is installed in namespace flux-system

---
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: gitlab-agent
  namespace: flux-system
spec:
  chart:
    spec:
      chart: gitlab-agent
      version: "2.11.0"
      reconcileStrategy: ChartVersion
      sourceRef:
        kind: HelmRepository
        name: gitlab
        namespace: flux-system
  install:
    createNamespace: true
  interval: 1m0s
  releaseName: gitlab-agent
  targetNamespace: flux-system
  values:
    config:
      kasAddress: wss://gitlab-forge.din.developpement-durable.gouv.fr/-/kubernetes-agent/
      secretName: gitlab-agent-token
    # -- CUSTOM VALUES --
      operational_container_scanning:
      # -- enables automatic RBAC creation for the operational container scanning feature
        enabled: false # Because not supported on Gitlab community edition and would create unnecessary stuffs
    # -- CUSTOM VALUES --
    # -- set number of replicas
    replicas: 1 # If running 2 replicas, gitlab-agent needs to have RBAC to get resource "leases" in API group "coordination.k8s.io" in the namespace "infra". And the view role doesn't have this built-in. As of now we don't need HA, so we set replica to 1 and with only 1 replica it doesn't acccess to resource leases !
    rbac:
      # -- Specifies whether RBAC resources should be created
      create: true
      # -- Set to a rolename to use existing role.
      # @default -- cluster-admin
      # useExistingRole: view # Read only roles for gitlab agent only because gitlab user impernation is not supported in Gitlabe community edition !

= same issue when gitlab-agent is cluster-admin.

 

  • gitlab-agent : I tested both v18.0.0 ( latest helm chart ) and v17.8.0 ( chart 2.11.0 )

 

I found issues/279 that dates back from some time ago.

Any way to fix this ? Or I’m left with configuring the commit event to fluxcd receiver manually ? Any help appreciated !!