PagerDuty

A PagerDuty webhook delivers incident events from a PagerDuty V3 webhook subscription into Plural. Each event is one alert, keyed by the PagerDuty incident, so later events for the same incident update it. PagerDuty generates the signing secret, and you paste it into Plural.

An alert does not start a job by itself. A job starts only when a webhook trigger on a workbench matches the alert, and only when the incident starts firing: the first event for an incident, or an event that reopens a resolved incident. An acknowledgment or another update to an open incident does not start another job, and an incident starts at most one job per workbench per hour.

What Plural reads from the payload

PagerDuty sends one event per request, with the incident under event.data:

{
  "event": {
    "id": "5ac64822-4adc-4fda-ade0-410becf0de4f",
    "event_type": "incident.triggered",
    "data": {
      "id": "PGR0VU2",
      "type": "incident",
      "html_url": "https://acme.pagerduty.com/incidents/PGR0VU2",
      "title": "Checkout latency high (Plural Cluster: prod-us-east)",
      "status": "triggered",
      "priority": { "summary": "P1" }
    }
  }
}

• data.id identifies the alert, falling back to event.id.
• data.status sets the state. resolved is resolved. Everything else, including triggered and acknowledged, is firing.
• data.title becomes the alert title, and data.html_url the link back to PagerDuty.
• data.priority.summary sets the severity: P1 is critical, P2 high, P3 and P4 medium, and P5 low. A missing priority or another priority name leaves the severity undefined.
• data.summary, when present, becomes the alert summary. Otherwise the summary reads "No summary available".
• data.custom_details, when present, is stored as tags.

Associate alerts with Plural resources

Plural keeps an alert only when it belongs to a Plural project, cluster, or service, or when a workbench trigger matches it. Otherwise the event is acknowledged and ignored.

Plural first reads plrl_project (the project name), plrl_cluster (the cluster handle), and plrl_service (the service name on that cluster, together with plrl_cluster) from data.custom_details. For any of the three it does not find there, it looks for markers in the incident title and summary: Plural Project: <name>, Plural Cluster: <handle>, and Plural Service: <name>, or the forms plrl-project-<name>, plrl-cluster-<handle>, and plrl-service-<name>. A service marker also needs the cluster. Putting a marker in the incident title, for example from the alert that opens the incident, is the most direct way to associate it.

A workbench trigger matches against the alert title and summary, not the raw payload. When several triggers on the webhook match, the one with the highest priority wins.