Grafana

A Grafana webhook delivers alert notifications from a Grafana webhook contact point into Plural. Each notification can carry several alerts, and Plural stores each one as its own alert, both when it fires and when it resolves.

An alert does not start a job by itself. A job starts only when a webhook trigger on a workbench matches the alert, and only when the alert starts firing: the first firing notification for an alert, or a firing notification for an alert that had resolved. Repeat notifications for an alert that is still firing do not start another job, and an alert starts at most one job per workbench per hour.

What Plural reads from the payload

Plural reads Grafana's default webhook payload:

{
  "title": "[FIRING:1] High memory usage (blue)",
  "message": "**Firing** ...",
  "commonLabels": { "team": "blue" },
  "alerts": [
    {
      "status": "firing",
      "fingerprint": "c6eadffa33fcdf37",
      "generatorURL": "https://grafana.example.com/alerting/1afz29v7z/edit",
      "labels": {
        "alertname": "High memory usage",
        "plrl_cluster": "prod-us-east",
        "plrl_service": "checkout"
      },
      "annotations": {
        "summary": "Memory above 90% on checkout",
        "severity": "high"
      }
    }
  ]
}

• Each entry in alerts becomes one alert. commonLabels and commonAnnotations are merged into every entry.
• fingerprint identifies the alert, so a later notification for the same fingerprint updates it.
• status sets the state. firing is firing, anything else is resolved.
• The top-level title becomes the alert title, falling back to the alertname label.
• The summary annotation and the top-level message become the alert summary.
• generatorURL is the link back to Grafana.
• The severity annotation sets the severity when it is low, medium, high, or critical. Any other value, or a severity label instead of an annotation, leaves the severity undefined.
• Labels with string values are stored as tags.

Plural stores the whole payload with the alert. The workbench alert details show the evaluated query values from values and a link to Grafana's silenceURL.

Associate alerts with Plural resources

Plural keeps an alert only when it belongs to a Plural project, cluster, or service, or when a workbench trigger matches it. Otherwise the notification is acknowledged and ignored. Set these labels on the alert rule:

• plrl_project: the project name.
• plrl_cluster: the cluster handle.
• plrl_service: the service name on that cluster. It only applies together with plrl_cluster.