Judges

Prompt-based LLM evaluators that score live traces. Three response shapes: binary, classification, score.

A judge is a configurable LLM evaluator. You give it a prompt — possibly with {{variables}} rendered from a trace — and it returns a typed verdict you can attach to traces, gate on, or alert from. Judges live server-side and run through Polarity's model proxy.

Judges are the building block underneath Behaviors. A behavior is a UI-friendly wrapping of one judge with categorical labels; the judge itself is the raw scoring primitive.

Three judge types

Pick the response shape that matches the question you're asking.

Type	Returns	Use for
`binary`	`{passed: bool, reason: string}`	Pass/fail checks: "Did the agent give a clear refund timeline?"
`classification`	`{category: string, reason: string}`	Multi-label routing: "Sentiment: positive / neutral / negative"
`score`	`{score: number, reason: string}`	Continuous rating in a bounded range, e.g. `[0, 1]` or `[1, 5]`

classification is also accepted as the alias categorical and score as numeric in SDK calls.

Create a judge

Open Judges in the sidebar of app.polarity.so → New Judge. Fill in:

Name — short identifier (Helpful Refund Response)
Type — binary / classification / score
Prompt — natural language, with {{variable}} slots for trace fields
Model — defaults to polarity/paragon-fast; pick paragon-md or paragon-max for harder calls
Variables — the keys you reference in the prompt ({{user_message}}, {{agent_response}}, etc.)
Categories (classification only) — list of allowed labels with descriptions

Save. The judge is immediately callable via /v1/judges/{id}/evaluate.

Run a judge

Once a judge exists, hit POST /v1/judges/{id}/evaluate with the variable values. The server renders the template, calls the configured model through paragon-llm-proxy, parses the verdict, and records it.

verdict = plr.judges.evaluate(
    judge.id,
    inputs={
        "user_message": "I want a refund for order #9384",
        "agent_response": "I'll get back to you on that.",
    },
)
 
print(verdict.passed)   # False
print(verdict.reason)   # "No timeline given, no actionable next step."

Prompt templating

Variable refs use {{name}} syntax. Unknown refs are left visible so the model can flag them.

The user said: {{user_message}}
The agent replied: {{agent_response}}
Reference policy: {{policy_doc}}    ← unresolved refs stay as-is

Variables can be any string. If you need structured input (a whole trace, JSON), JSON-encode it before passing:

plr.judges.evaluate(judge.id, inputs={
    "trace_json": json.dumps(trace.to_dict()),
})

Evaluation settings

Each judge carries an evaluation_settings JSON block controlling when and how it auto-runs:

{
  "mode": "on_demand" | "continuous",
  "sampling_rate": 1.0,
  "requires_vars": true
}

mode: continuous — the judge fires on every incoming trace that matches its variable shape
sampling_rate — fraction of matching traces to score in continuous mode (0–1)
requires_vars — when true, traces missing any of the declared variables are skipped

Set these via the dashboard or plr.judges.update(id, evaluation_settings={...}).

Inter-rater alignment

Each judge tracks an optional alignment score (0–1) representing how often it agrees with human reviewers on a labeled set. Higher is better. Calibrate by sampling a few hundred traces and labeling them in the dashboard — the alignment score updates as labels accumulate.

A judge with alignment < 0.7 is probably noisy and shouldn't gate alerts yet — iterate on the prompt first.

Picking a model

evaluate calls route through Polarity's model proxy — judge usage is included in the flat Polarity contract. Pick paragon-fast for high-volume continuous scoring (the default for new behaviors); reserve paragon-max for hard calls where alignment matters more than throughput.

Next steps

Behaviors — wrap a judge with named categorical options so dashboards and automations can filter on its output
Paragon Agent — the in-product AI that can sample your traces and propose a judge prompt for you
Alerts — fire Slack / email / dataset-add when a judge verdict crosses a threshold

Behavior gallery LLM-as-Judge