Agent Monitoring
Tracing + behaviors + judges + Slack — the closed loop for catching production agent issues without writing eval code.
Agent monitoring is the headline use case for Polarity. The loop is four primitives in series:
Tracing
Wrap your LLM client with plr.wrap(). Every model call + every tool call streams into Polarity automatically.
Behaviors
Name a thing you care about in plain English — "was this reply helpful?" Every new trace gets the label within 10s.
Judges
The scoring primitive a behavior wraps. Reusable LLM-as-judge configs, versioned, runnable from anywhere.
Slack + Alerts
@Polarity investigates from any Slack thread. Behavior-fail rate spikes → alert lands in your on-call channel.
The 5-minute setup
# 1. Install
npm install @polarityinc/polarity # or pip install polarity-sdk
# 2. Drop your key
echo 'POLARITY_API_KEY=plr_live_...' >> .env// 3. Wrap your LLM client — one line per provider
import { Polarity } from "@polarityinc/polarity";
import Anthropic from "@anthropic-ai/sdk";
const plr = new Polarity();
plr.initTracing();
const anthropic = plr.wrap(new Anthropic());
// 4. Use it normally — every call is now traced.
await anthropic.messages.create({ model: "claude-sonnet-4-5", ... });Open app.polarity.so/app/polarity/traces — your traces are flowing.
Now create a behavior in the dashboard ("did the agent give a clear refund timeline?"), invite @Polarity to a Slack channel, and you're running the full loop.
What you get
| Capability | Page |
|---|---|
| Every LLM + tool call traced, with token + cost + latency | LLM Tracing |
| Custom spans for non-LLM work (DB queries, HTTP, etc.) | Custom Spans |
| Production mode — high-cardinality tracing optimized for live traffic | Production Mode |
| 0-config tracing for installed provider SDKs | Auto-Instrument |
| Named labels on every trace, no eval code required | Behaviors |
| 12 example behaviors with full prompts | Behavior gallery |
| Versioned LLM-as-judge configs reusable across behaviors and alerts | Judges |
| 28 built-in scorers (heuristic, LLM, RAG, embedding) | Scorers Library |
| Real-prod-trace → versioned dataset → re-eval after the fix | Datasets |
| Threshold + behavior-based alerts to Slack / webhook / PagerDuty | Alerts |
@Polarity in Slack channels — investigate without leaving the conversation | Slack |
| In-product AI that proposes judges + investigates traces | Paragon Agent |
The closed loop
The four primitives compose into a single workflow:
Production trace ──▶ Behavior labels it ──▶ Alert fires ──▶ Slack ping
│ │
│ ▼
│ Engineer investigates
│ in-thread with @Polarity
│ │
│ ▼
│ Add failing traces to dataset
│ │
▼ ▼
…keeps flowing… Ship the fix, re-eval
Every step is one page in this section. The dashboard surfaces the loop visually so you don't have to remember the wiring.
Why this beats homegrown observability
Most teams roll their own with OTel + a custom dashboard + a half-written grader script. That gets you traces but stops short of named, scored, alertable labels on each trace. Polarity makes the loop turnkey:
- No eval-code-per-question. A behavior is one English sentence, scored by a versioned judge.
- No regression rot. Failing traces become a dataset automatically; the next deploy re-evals against them.
- No context-switching during incidents.
@Polarityis in the Slack thread where you're already debugging. - Versioned everything. Prompts, judges, datasets, behaviors — all versioned. Old experiments stay reproducible after edits.
Already have OTel?
Polarity accepts OTel out of the box — point your OTLP exporter at https://api.polarity.so/v1/otel and your spans show up alongside SDK-emitted traces. See OpenTelemetry.
Next
- Tracing — wrap your first LLM client (one line).
- Behaviors — define your first scored label.
- Behavior gallery — copy-paste-able prompts for the most common behaviors.
- Slack — install
@Polarityin your team channel.