Webhook Ingest: How to Turn Any Observability Tool Into an Incident Source

Datadog can tell you that p99 latency on the payment API just doubled. Prometheus can tell you the error rate crossed a threshold. Sentry can tell you a new exception is spiking after a deploy. None of those tools is where you want to run the rest of the incident.

On-call paging, escalation, the shared timeline, status page updates, and the postmortem all need one workflow. Generic webhook ingest is how detection wherever it happens becomes response in one place.

What generic webhook ingest is

Your incident management tool exposes an HTTP endpoint. Observability tools send an authenticated POST when an alert fires. The incident tool parses the payload, opens an incident with severity and service context, and runs the normal routing path.

That is the whole idea. No custom agent required for every source. If a system can emit a webhook, it can become an incident source.

Why observability tools alone are not enough

Detection and incident management solve different problems.

Observability products are built to evaluate metrics, logs, and traces, then fire alerts. Incident management is built to interrupt the right person, coordinate responders, communicate outward, and leave a record you can learn from. When each alert source pages through its own path, you get split brain. One person is in Datadog, another is in Sentry, and nobody owns the customer update.

Webhook ingest keeps the best detector for each signal and still funnels response into a single operating rhythm.

How it works technically

The incident tool provides an ingest URL and an authentication method, usually an API key header or HMAC signature over the body. You configure the observability tool's webhook or notification channel to point at that URL.

When a payload arrives, the ingest layer validates auth, maps fields to an incident model, and opens or updates an incident. Severity drives priority. Service or team labels drive routing. The alert title and condition become the incident summary. From there, escalation policies, stakeholder notifications, and status page workflows proceed as they would for any other incident.

If the source later sends a resolved event, good implementations update or auto-resolve the incident so humans are not closing duplicates by hand.

Configuring common sources

Datadog. Use the Webhooks integration. Create a webhook that posts to your ingest URL with the auth header your incident tool expects. On monitors, add that webhook as a notification target with @webhook-your-name. Keep monitor tags like service and env consistent so the ingest mapper can route correctly. Datadog's event payload includes the monitor name, query, and snapshot links. Map those into title, condition, and deep link fields.

Prometheus Alertmanager. Add a webhook_configs receiver and route critical alerts to it. Put severity and service in labels, not only in annotations, because labels are what most routers key on.

receivers:
  - name: incident-tool
    webhook_configs:
      - url: "https://example.com/ingest/alertmanager"
        send_resolved: true
        http_config:
          authorization:
            credentials: "YOUR_INGEST_TOKEN"

Grafana. Create a contact point with type webhook. Point it at the ingest URL, add the auth header, and attach that contact point to notification policies for production alert rules. Use labels for severity and service the same way you would in Alertmanager so routing stays predictable across sources.

Sentry. Create an issue alert rule with a webhook action when an issue is first seen or regresses in production. Include environment and release in the payload. Sentry is noisy if you page on every event, so gate the rule on frequency or a production environment tag before it hits incident ingest.

Payload shape and field mapping

A good payload is boring and complete. Severity, service name, alert title, triggering condition, environment, and a URL back to the source. Everything else is optional detail.

Alertmanager-style fields often look like this.

{
  "status": "firing",
  "commonLabels": {
    "alertname": "PaymentAPIHighLatency",
    "severity": "critical",
    "service": "payments-api",
    "env": "prod"
  },
  "commonAnnotations": {
    "summary": "p99 latency above 800ms for 5 minutes",
    "runbook_url": "https://runbooks.example.com/payments-latency"
  }
}

Map severity to incident priority, service to ownership and routing, summary or monitor name to the title, and runbook_url into the incident details. Datadog monitor webhooks differ in envelope shape, but the same fields should exist somewhere in tags and message text. If they do not, fix the monitor before you blame the ingest parser.

The failure mode I see most is empty service labels. The webhook succeeds, an incident opens, and it sits in a global queue because nothing matched a routing rule.

The two-way flow

One-way ingest is enough to start. Better setups close the loop. When the incident resolves, the incident tool calls back to acknowledge or resolve the alert in the source system where the API allows it. Alertmanager gets send_resolved traffic. Some vendors accept an event update to clear the firing state.

Two-way flow prevents the awkward state where production is fixed, the incident is closed, and the observability UI still screams until someone clicks around. It is optional. Getting one-way ingest solid with correct labels is the higher-leverage first step.

Deduplication belongs in that same design conversation. A single failing dependency can fan out into Datadog, Prometheus, and Sentry webhooks within a minute. If every payload opens a fresh incident, on-call drowns in siblings. Prefer grouping keys based on service plus alert name, or attach follow-on alerts to an open incident when the fingerprint matches. Exact mechanics vary by tool. The operating rule does not. One user-facing failure should produce one incident of record whenever you can manage it.

Where Vigiles fits

Generic webhook ingest is on the near-term Vigiles roadmap so Datadog, Prometheus, Grafana, Sentry, New Relic, and other webhook-capable tools can open Vigiles incidents with the same routing, escalation, and postmortem path as native monitors. Until that ships, the design above is still the right architecture. Centralise response even when detection stays distributed.

If an alert can POST JSON, it should be able to page the on-call through one incident system. Everything after the alert is common work. Stop reinventing that part in every observability console.