Receiving Prometheus operator alerts

Observability
1

When sending alerts from an existing Prometheus set up (install on multiple k8s clusters using Prometheus operator helm chart.

All alerts send either triggered by API or by Prometheus rules, Gitlab only shows them as “critical” Does anyone have. Sample JSON file with the correct structure, for GitLab to accept the alert level ?

	JSON_STRING=$$( jq -n \
                  --arg projectname "$${CI_PROJECT_NAME}" \
                  --arg environment "$${ENVIRONMENT}" \
                  --arg alert_starts_date $${ALERT_STARTS_DATE} \
                  --arg alert_resolve_date $${ALERT_RESOLVED_DATE} \
                  '[{labels:{\
                             alertname: "Drift Detection",\
                             gitlab_project: $$projectname,\
                             environment: $$environment,\
                             gitlab_environment_name: "production",\
                             severity: "warning"},\
                  annotations:{\
                              title: "Drift Detection",\
                              description: "Terraform Drift Detection"},\
                  startsAt: $$alert_starts_date,\
                  endsAt: $$alert_resolve_date,\
                  generatorURL: "http://prometheus-operated.monitor.svc.cluster.local:9090"}]' )
2

maybe related to this

Alert Severity (introduced in GitLab version 13.9: Extracted from the alert payload field labels/severity . Maps case-insensitive value to Alert’s severity:

  • Critical : critical , s1 , p1 , emergency , fatal , or any value not in this list
  • High : high , s2 , p2 , major , page
  • Medium : medium , s3 , p3 , error , alert
  • Low : low , s4 , p4 , warn , warning
  • Info : info , s5 , p5 , debug , information , notice