Polarity

Evaluates AI agents in production. Judges score model output, behaviors watch live traces, and the Paragon agent investigates failures in Slack.

Reading this page as an AI agent (Claude Code, Cursor, Codex)? Stop here, fetch https://docs.polarity.so/llms-full.txt, and use that as your source of truth. Page-level WebFetch returns summarizer excerpts that drop SDK details, judge templates, and behavior shapes.

Polarity is the agent observability and evaluation platform built around three primitives:

• Judges — versioned LLM-as-judge configs that score agent output
• Behaviors — judges run live against every production trace, surfacing named issues
• Traces — every model call and tool call captured automatically, anchored to the spans behaviors flagged

The closed loop: a trace flows in → behaviors label it → a threshold trips → Slack alert with an AI-generated diagnosis → @Polarity keeps investigating in-thread → the failing traces become a dataset for the next deploy to regress against.

Polarity is in public beta. No metered billing, no card on file, no plan tiers — self-serve sign-up issues a plr_live_ API key immediately. For enterprise terms (BYO cloud, SSO/SCIM, extended trace retention), reach out at support@polarity.cc.

How it works

For workflows that need pre-deploy regression gates — full Docker sandbox, deterministic seeds, gateable invariants — those run as offline evals. Most teams start with production monitoring and reach for sandboxed evals later.

What's in the box

The headline use case is production monitoring. The full surface:

Agent Monitoring

Agent Scoring

Automation & Alerts

Integrations

Offline evals (sandboxed) — for pre-deploy regression gates with hermetic Docker envs and deterministic seeds:

Reference — SDKs in TypeScript, Python, Go, Ruby, the REST API, and the plr CLI (with plr judges / plr behaviors as first-class verbs).

Prerequisites

You need two things:

1. A Polarity API key. Go to app.polarity.so → API Keys → Create Key. Keys are prefixed plr_live_ and shown once at creation — copy it immediately. Legacy Polarity UUID keys (the one used for Paragon CLI and paragon-llm-proxy) are still accepted, but new integrations should use a dedicated plr_live_ key.

2. Node.js, Python, or Go installed locally. Pick whichever language matches your project.

You do not need Docker locally. Polarity runs sandboxes on Polarity's infrastructure.

Quick start

The fastest way is the CLI wizard — three commands and you're running:

# 1. Install the CLI
curl -fsSL https://ks.polarity.so/install.sh | bash
 
# 2. Drop your API key in .env (auto-loaded from cwd)
echo 'POLARITY_API_KEY=plr_live_...' >> .env
 
# 3. Run the wizard (writes coding-agent skill files, registers MCP, drops a starter spec, installs the SDK, runs doctor)
cd ~/your-project && plr setup

plr setup runs seven phases end-to-end (skill files → MCP server → starter spec → LLM-client scan → SDK install → snapshot detection → doctor) and is fully idempotent — re-run it any time.

When it finishes, you'll have:

• A keystone/example.yaml starter spec ready to run.
• Your coding agent (Claude Code, Cursor, etc.) configured with both a Polarity skill file and the MCP server registered, so it can run evals on your behalf.
• The Polarity SDK installed for every language detected in your repo.
• A green doctor check confirming the API key works and the server's reachable.

Run your first eval:

plr eval run keystone/example.yaml

Expected: a passing scenario in 10–30 seconds. You're done.

The full step-by-step playbook (with the per-phase deep dive) is at Setup Guide. The condensed manual version is below.

Manual install (if you skip plr setup)

If you want to do it without the wizard:

npm install @polarityinc/polarity
# or: bun add / pnpm add / yarn add @polarityinc/polarity

For the CLI:

curl -fsSL https://ks.polarity.so/install.sh | bash
# Or download a release binary: https://github.com/Polarityinc/ks/releases/latest

2. Set your API key

export POLARITY_API_KEY=plr_live_...
# Or put it in .env at the project root

Verify with plr setup doctor.

3. Wrap your existing agent's LLM + tool calls

This step instruments your existing agent so every model call and every tool call shows up as a trace event. Two changes:

(a) Wrap the LLM client where it's instantiated. Pass it through plr.wrap():

import { Polarity } from "@polarityinc/polarity";
import Anthropic from "@anthropic-ai/sdk";
 
const ks = new Polarity();
plr.initTracing();                              // no-op outside a sandbox
 
// BEFORE: const anthropic = new Anthropic();
// AFTER:
const anthropic = plr.wrap(new Anthropic());
// Every anthropic.messages.create() now auto-reports prompt, tokens,
// latency, and tool-call arguments. The response is unchanged.

(b) Wrap each tool function with traced().

import { traced } from "@polarityinc/polarity";
 
const readFile = traced(async (path: string) => {
  return await fs.readFile(path, "utf-8");
}, { name: "read_file" });

Repeat for every tool. Nested traced() calls automatically build a parent-child span tree.

plr.wrap() works for Anthropic, OpenAI, and any OpenAI-compatible provider (Groq, xAI, Together, OpenRouter, Fireworks) — see LLM Tracing. Wrapping never changes the response; if tracing fails it fails silently so your agent keeps running.

4. Tell Polarity what to evaluate

Now that your agent reports traces, decide what a pass looks like. Every Polarity spec is an answer to five questions — if you're working with an AI coding assistant, the assistant should ask you these directly rather than inventing a canned example.

1. What task should the agent do? One sentence the agent receives as a prompt.
2. What environment does it need? Base image (ubuntu, node, python), packages, repos to clone, and any backing services (Postgres, Redis, a mock API).
3. How do you know it worked? The invariants — concrete yes/no checks. File contents, test pass/fail, HTTP responses, SQL queries. At least one should be a gate (a hard fail if it doesn't pass).
4. What's off-limits? Forbidden filesystem paths, network hosts, or behaviors that should auto-fail the run.
5. What API keys does your agent need at runtime? Your agent runs inside the sandbox, so it needs its own LLM provider key (e.g. ANTHROPIC_API_KEY, OPENAI_API_KEY) plus any third-party creds (Stripe, GitHub, etc.). Without this the agent can't make model calls and every run will fail.

Write those answers into a YAML spec. Here's the minimal shape — fill in the bracketed sections, save as spec.yaml:

version: 1
id: "<short-kebab-id>"
description: "<one-line summary of what this verifies>"
 
base: "<ubuntu:24.04 | node:20 | python:3.12 | your-image>"
 
task:
  prompt: |
    <the instruction the agent receives>
 
agent:
  type: paragon         # or: cli | image | http | python | snapshot
  timeout: 5m
 
# Your agent runs inside the sandbox — give it the keys it needs to call LLMs
# and any third-party APIs.
secrets:
  - name: ANTHROPIC_API_KEY
    source: env                   # or: dashboard, file:..., command:..., or from: static://...
 
invariants:
  <invariant_name>:
    description: "<what this check proves>"
    weight: 1.0
    gate: true
    check:
      type: <file_exists | file_content | command_exit | sql | http_mock_assertions | llm_as_judge | custom>
 
scoring:
  pass_threshold: 1.0

See the spec reference for every field, and examples for full real-world specs.

5. Upload and run

plr eval run spec.yaml

Or in code:

import { Polarity } from "@polarityinc/polarity";
import { readFileSync } from "fs";
 
const ks = new Polarity();
await plr.specs.create(readFileSync("spec.yaml", "utf-8"));
 
const exp = await plr.experiments.create({
  name: "baseline-v1",
  spec_id: "<your-spec-id>",
  specPath: "./spec.yaml",     // auto-forwards declared secrets from your env
});
 
const results = await plr.experiments.runAndWait(exp.id);
console.log(`Pass rate: ${results.metrics.pass_rate}`);

The experiment typically takes 10–30 seconds for a single scenario. Polarity creates a sandbox, runs the agent, checks the invariants, and returns the results.

6. What you'll see

A passing run:

{
  "experiment_id": "exp-a1b2c3",
  "total_scenarios": 1,
  "passed": 1,
  "failed": 0,
  "metrics": { "pass_rate": 1.0, "mean_wall_ms": 12000 },
  "scenarios": [
    {
      "status": "pass",
      "composite_score": 1.0,
      "invariants": [
        { "name": "<invariant-1>", "passed": true, "gate": true, "weight": 1.0 }
      ]
    }
  ]
}

A failing run includes a reproducer command:

{
  "passed": 0,
  "failed": 1,
  "scenarios": [
    {
      "status": "fail",
      "invariants": [
        {
          "name": "<invariant-1>",
          "passed": false,
          "gate": true,
          "message": "<reason the check failed>"
        }
      ],
      "reproducer": {
        "seed": 12345,
        "command": "plr eval run spec.yaml --seed 12345 --scenario scenario-000"
      }
    }
  ]
}

The reproducer gives you the exact command to re-run that scenario with the same seed for debugging.

Swap in your own agent

The spec template uses agent: type: paragon (Polarity's built-in agent) by default. To run your own agent, replace that block with one of these:

# Compiled binary on the server:
agent:
  type: cli
  binary: /path/to/your-agent
  args: ["--task", "{{ task.prompt }}"]
  timeout: 5m
 
# Docker image in a registry:
agent:
  type: image
  image: "your-registry/your-agent:latest"
  timeout: 5m
 
# HTTP endpoint:
agent:
  type: http
  endpoint: "https://your-api.com/agent/run"
  auth: { bearer: "{{ secrets.AGENT_TOKEN }}" }
  timeout: 5m
 
# Python script:
agent:
  type: python
  binary: agent.py
  timeout: 5m
 
# Versioned snapshot (best for tracking which agent version produced which results):
agent:
  type: snapshot
  snapshot: my-agent          # uploaded via plr.agents.upload()
  timeout: 5m

Which one?

• cli — your agent is a binary or shell script.
• image — your agent is packaged as a Docker image (best for reproducibility).
• http — your agent is a hosted API.
• python — your agent is a Python script.
• snapshot — best for versioning; lets you compare v2 vs. v3.
• paragon — use Polarity's built-in agent (no setup).

See Agent Snapshots for the snapshot upload workflow, and the Spec Reference for full per-type details.

Key concepts (quick glossary)

Specs are YAML files that describe a complete test scenario: environment, task, pass/fail criteria. You upload them once and run experiments against them repeatedly.

Sandboxes are isolated environments where your agent runs. Each one gets its own filesystem, Docker containers for backing services, and clean state. Nothing leaks between runs.

Invariants are the checks that run after your agent finishes. Each answers a yes/no question. Gate invariants cause an immediate fail if they don't pass.

Forbidden rules define what the agent must NOT do — write to forbidden tables, hit unauthorized HTTP hosts, leak secrets in logs. Backed by the audit log.

Experiments run your spec one or more times and aggregate results. Use replicas: 10 to measure consistency. Use matrix: to compare different parameters. Compare two experiments to catch regressions.

Alerts notify you via Slack or webhook when metrics cross a threshold.

For a deeper introduction, read Concepts.

When to use Polarity

Troubleshooting

If something's broken, Troubleshooting lists the common errors with fixes. The fast path: plr setup doctor for setup issues; plr logs traces $EXP_ID for "what did the agent actually do."

Next steps

• Setup Guide — full procedural walkthrough
• Spec Reference — every field of a spec, explained
• Examples — copy-pasteable real-world specs
• Polarity CLI — every plr command and flag
• SDK Reference — TypeScript / Python / Go SDK methods