Alerts

Slack and webhook notifications when experiment metrics cross a threshold.

An alert fires when a metric crosses a threshold across one or more experiments. Notify a Slack channel, hit a webhook, or post via a Slack Bot — Polarity evaluates rules continuously as new experiments complete.

Anatomy

{
  name: "pass-rate-drop",
  eval_id: "fix-failing-test",         # optional — scope to one spec
  condition: "pass_rate < 0.8",
  window: "last_5_runs",               # optional — only the last N runs
  notify: "slack",
  slack_channel: "#agent-alerts",
}

Every alert has:

Conditions

Conditions are <metric> <operator> <value>:

Metrics:

Operators: <, <=, >, >=, ==, !=

Examples:

pass_rate < 0.8
mean_wall_ms > 30000
total_cost_usd > 5.00
side_effect_violations > 0
tool_success_rate < 0.9

Slack channel notifications

Post a Block Kit message to a Slack channel via Slack Bot:

await plr.alerts.create({
  name: "cost-spike",
  condition: "mean_cost_per_run_usd > 2.00",
  notify: "slack",
  slack_channel: "#agent-alerts",
});

Setup required: the Polarity server needs SLACK_BOT_TOKEN set in its environment (typically configured by Polarity for hosted users; self-hosted folks set it themselves). The bot needs chat:write on the channel; invite it manually if posting fails.

What lands in Slack

Behavior alerts post as a Block Kit message with:

• One-line summary — Helpful Refund Response: 12 of 18 traces failing (67%) over the last 1h
• Span card — the failing span anchored in the trace, with model + tool + latency + cost inline
• AI-generated diagnosis — a one-paragraph root-cause hypothesis from the Paragon Agent ("agent is misclassifying partial-refund requests as full-refund because the rubric…")
• Deep-link to trace — opens the full timeline in the dashboard
• Copy debug context button — copies the trace summary + spans + recent behavior verdicts as plain-text JSON, paste-able into Claude / Cursor / your scratchpad

The diagnosis + debug bundle is the killer: the on-call engineer pastes it into a coding assistant and starts debugging without context-switching.

Behavior threshold alerts

For behaviors, the threshold lives on the Behavior itself, not in a separate alert rule. Set baseline_pct and baseline_direction on the behavior and the runner computes a rolling detection rate over the last 50 verdicts. The alert fires edge-triggered when the rate crosses the threshold, and a server-side latch prevents re-firing while still breached — so a single false reading doesn't page anyone, but a sustained drift does:

// Set the threshold on the behavior (e.g. via plr.behaviors.update).
// Once set, the runner alerts whenever the rolling fail-rate crosses it.
await plr.behaviors.update(behaviorId, {
  baseline_pct: 0.2,         // 20% fail-rate threshold
  baseline_direction: "above", // fire when above; also: "below" or "either"
});

Fixed-window mechanics:

• Window: last 50 verdicts (BaselineWindowN).
• Min sample: 20 verdicts (BaselineMinSample) — fewer than this and the runner stays silent.
• Edge-triggered: alert fires once when the rate crosses into the breached side; the latch resets when the rate recovers.

Routing for the resulting notification (Slack channel, webhook) is configured at the behavior / workspace level, not per-rule.

Webhook notifications

POST a JSON body to any URL:

await plr.alerts.create({
  name: "pass-rate-drop",
  eval_id: "fix-failing-test",
  condition: "pass_rate < 0.8",
  notify: "webhook",
  webhook_url: "https://hooks.slack.com/services/T00/B00/xxx",
});

The body shape:

{
  "rule_id": "alert_abc",
  "rule_name": "pass-rate-drop",
  "reason": "pass_rate dropped to 0.65 (threshold 0.8)",
  "experiment_id": "exp-xyz",
  "experiment_name": "ci-2026-04-28",
  "fired_at": "2026-04-28T22:00:00Z",
  "metric_value": 0.65,
  "threshold": 0.8
}

Slack incoming-webhook URLs

Auto-detected and translated to Slack Block Kit messages. Slack's incoming-webhook URLs (https://hooks.slack.com/services/...) get a richer rendered card; other URLs receive the raw JSON above.

// Slack incoming-webhook URL → rich Block Kit message
await plr.alerts.create({
  name: "...",
  condition: "...",
  notify: "webhook",
  webhook_url: "https://hooks.slack.com/services/T00/B00/xxx",
});
 
// Any other URL → raw JSON POST
await plr.alerts.create({
  name: "...",
  condition: "...",
  notify: "webhook",
  webhook_url: "https://my-monitoring-system.example.com/polarity-events",
});

Listing and deleting

// List
const all = await plr.alerts.list();
// AlertRule[]
 
// Delete
await plr.alerts.delete("alert_abc");

GET /v1/alerts and DELETE /v1/alerts/<id>.

Evaluation

Alerts are evaluated server-side after every experiment completes:

1. Find every alert rule whose eval_id matches (or is null — global rules).
2. For each, fetch the relevant experiment's metrics.
3. If condition is true, fire the notification.
4. Webhook calls retry up to 3 times with exponential backoff on 5xx / network errors.

There's no rate limiting per rule — if 100 experiments fail in an hour, you get 100 notifications. Use window: to debounce:

{
  name: "pass-rate-drop",
  condition: "pass_rate < 0.8",
  window: "last_5_runs",     # only fire if the LAST 5 runs all crossed the threshold
  notify: "slack",
  slack_channel: "#agent-alerts",
}

Patterns

CI-noise dampening

Don't alert on every flaky run — only on sustained drops:

{
  name: "ci-pass-rate-degradation",
  eval_id: "ci-regression",
  condition: "pass_rate < 0.85",
  window: "last_5_runs",
  notify: "slack",
  slack_channel: "#eng",
}

Cost alarm

{
  name: "cost-alarm",
  condition: "mean_cost_per_run_usd > 1.00",
  notify: "slack",
  slack_channel: "#oncall",
}

A run that suddenly costs 5× normal is usually a bug — an infinite loop, a spec change, a tool that's calling itself. Alarm fast.

Latency regression

{
  name: "latency-regression",
  condition: "p95_wall_ms > 60000",
  notify: "webhook",
  webhook_url: "https://datadog.com/webhook/...",
}

Pipe to Datadog/PagerDuty/whatever your team uses for ops.

Forbidden-rule tripwire

{
  name: "secrets-leak-tripwire",
  condition: "side_effect_violations > 0",
  notify: "slack",
  slack_channel: "#security",
}

Forbidden-rule violations are always worth a human's attention — fire on the first one.

Limits