Behaviors

UI-first scoring primitive. Name a thing you care about in plain English, see it scored on every trace.

A behavior is the thing you actually care about, scored on every trace, named in plain English. "Was the refund response helpful?" "Did the agent hallucinate an order number?" "Was this conversation frustrated or patient?"

Behaviors are how you turn a stream of traces into a stream of labeled, filterable, alertable signals — without writing eval code.

How a behavior relates to a judge

A behavior wraps exactly one judge and adds:

A flavor (binary or classifier) — must match the underlying judge type
Options with per-label metadata (display name, description, alert-worthy categories)
An evaluation mode (continuous, scheduled, manual)
A sampling rate for cost control
Span triggers that decide which trace spans this behavior runs against
Session scoring to extend a behavior beyond a single span to a whole conversation

The 1:1 mapping is enforced at the database layer. Behaviors are the named output; judges are the raw scoring primitive.

Create a behavior

The fastest path. Open Behaviors in the sidebar → New Behavior.

The form auto-creates the backing judge for you:

Field	What to put
Name	`Helpful Refund Response`
Description	One-liner customers will see in the dashboard
Flavor	`Binary` (yes/no) or `Classifier` (multi-label)
Prompt	One sentence describing what you're scoring
Model	`polarity/paragon-fast` (default)
Options (classifier only)	Add named labels — e.g. `frustrated`, `neutral`, `patient`
Evaluation mode	`Continuous` for live scoring
Sampling rate	`0.5` by default (score half of matching traces). Set `1.0` to score every trace, lower for cost control on high-volume agents.

Save. New traces flowing through /v1/traces get a label chip within ~10s.

Evaluation modes

Mode	What it does
`continuous`	Scores every matching trace as it lands. The only mode the runtime currently acts on — set this for production monitoring.
`scheduled`	Accepted by the SDK type, not yet runtime-active. Behaviors stamped `scheduled` are stored but skipped by the runner today.
`manual`	Accepted by the SDK type, not yet runtime-active. Will fire on explicit invoke when the runner gains the mode.

The Paragon Agent's create_behavior tool additionally accepts on_demand as an alias for the not-yet-active path.

Switch via the dashboard or plr.behaviors.update(id, evaluation_mode="continuous").

Span triggers

Behaviors don't have to score every span — they can target a subset.

Trigger	Scopes to
`all`	Every span in every trace
`errors`	Only spans with an exception or non-zero exit
`specific`	Match by `span_name` or attribute filter (configured per behavior)

For specific, set the filter via the behavior detail page. JSON filter shape mirrors trace-search filters.

Sampling rate

sampling_rate between 0 and 1 controls what fraction of matching spans get scored. New behaviors default to 0.5 — score half. Bump to 1.0 if you want a label on every trace; drop to 0.1 for trend detection on very high-volume agents.

Rule of thumb:

Trace volume	Sampling rate
< 100/day	`1.0`
100 – 10k/day	`0.5 – 1.0`
> 10k/day	`0.05 – 0.2`

Behavior stats

Every behavior surfaces three views on its detail page:

Detection rate — % of scored traces that triggered each option
28-day activity — sparkline of daily option counts
Per-option breakdown — drill into one label to see traces and patterns

These are computed live by /v1/behaviors/{id}/stats from keystone_traces — no nightly job, no stale data.

Session scoring

By default a behavior scores individual spans. Flip session_scoring to enabled and it'll also produce a session-level verdict across all spans tied to the same session_id. Useful for behaviors that depend on conversation flow ("Did the agent ever apologize?") rather than a single response.

Behavior labels = automation triggers

Once a behavior is running, Alerts can fire on its output:

WHEN  behavior "Helpful Refund Response" = false
AND   trace.llm_cost > $0.01
THEN  Slack #agent-alerts
AND   Add to dataset "regression-suite"

That's the closed loop: production trace → behavior label → automation → regression dataset → re-eval after the fix.

Behaviors vs scorers — when to use which

If you want…	Use
One sentence, scored on every prod trace, no code	Behavior
Heuristic checks (exact match, JSON diff, levenshtein)	Scorers library
Reference-based RAG scoring (faithfulness, context recall)	Scorers library
Sandbox-execution invariants (file exists, command exits 0)	Invariants

Behaviors are for live production scoring you defined in English. The other primitives are for offline eval-time scoring you defined in code.

Next steps

Judges — the scoring primitive a behavior wraps
Paragon Agent — ask the in-product AI to suggest behaviors based on your actual traces
Alerts — turn behavior labels into Slack pings + dataset additions

Auto-Instrument Behavior gallery