Gitlab Omnibus CE + Prometheus + Grafana Kubernetes Integration

Observability
,
1

Hi!
How to Gitlab Omnibus CE + Prometheus + Grafana Kubernetes Integration!?

I’d like to make Kubernetes cluster monitoring with GitLab CE integrated Prometheus and Grafana.

Prometheus and Grafana work well. Kubernetes are integrated with GitLab. But I don’t understand how is possible enable and configure their cooperation. Helm work. I read in GitLab FAQ Project integrations | GitLab

I deployed Prometheus into the gitlab-managed-apps but I see error.

I read these lines in the documentation:

To configure a resource to be monitored by Prometheus, simply set the following Kubernetes annotations:

  • prometheus.io/scrape to true to enable monitoring of the resource.
  • prometheus.io/port to define the port of the metrics endpoint.
  • prometheus.io/path to define the path of the metrics endpoint. Defaults to /metrics .

Where is it need this change? Maybe do you know the detailed instructions? Step by step.

prometheus error pod has unbound immediate PersistentVolumeClaims

{
  "kind": "PersistentVolumeClaim",
  "apiVersion": "v1",
  "metadata": {
    "name": "prometheus-prometheus-server",
    "namespace": "gitlab-managed-apps",
    "selfLink": "/api/v1/namespaces/gitlab-managed-apps/persistentvolumeclaims/prometheus-prometheus-server",
    "uid": "117cf105-df9a-11e9-9bd6-00155d011b19",
    "resourceVersion": "45005673",
    "creationTimestamp": "2019-09-25T13:40:42Z",
    "labels": {
      "app": "prometheus",
      "chart": "prometheus-6.7.3",
      "component": "server",
      "heritage": "Tiller",
      "release": "prometheus"
    },
    "finalizers": [
      "kubernetes.io/pvc-protection"
    ]
  },
  "spec": {
    "accessModes": [
      "ReadWriteOnce"
    ],
    "resources": {
      "requests": {
        "storage": "8Gi"
      }
    },
    "volumeMode": "Filesystem",
    "dataSource": null
  },
  "status": {
    "phase": "Pending"
  }
}

Ok I resoved Prometheus:

kubectl create clusterrolebinding --user system:serviceaccount:gitlab-managed-apps:prometheus-kube-state-metrics default-gitlab-prometheus-sa-admin --clusterrole cluster-admin

and

executed yaml

apiVersion: v1
kind: PersistentVolume
metadata:
  name: hostpatheightgigs
spec:
  capacity:
    storage: 8Gi
  volumeMode: Filesystem
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Recycle
  hostPath:
      path: "/k8s/data1"

All Kubernetes Prometheus PODs Statuss become OK.
I see metrics in indoor prometheus metric POD http://10.44.0.126:8080/metrics
and I see Prometheus server UI http://10.44.0.131:9090/grapfs

But I don’t see any new metrics from Kubernetes cluster in GitLab Prometheus?!
How does it have to work and see the result in GitLab? I didn’t find any video where I can see it.