Eval() primitive

Braintrust-parity Eval(name, { data, task, scores }) — the lightweight entry point for local dev and CI scoring.

Eval() is the simplest way to score a batch of inputs through your agent. You hand it data, a task function, and scorers; it runs the task per row in parallel, scores each output, and aggregates per scorer.

If POLARITY_API_KEY is set, the run is also reported to the dashboard as an experiment. If not, it runs purely locally — useful for fast iteration before committing to a spec.

Anatomy

import { Eval, Factuality } from "@polarityinc/polarity";
 
const result = await Eval("name", {
  data: [{ input, expected }, ...],
  task: async ({ input }) => myAgent(input),
  scores: [new Factuality()],
});

Minimal example

import { Eval, Factuality } from "@polarityinc/polarity";
import OpenAI from "openai";
 
const openai = new OpenAI();
 
const result = await Eval("summarize-quality", {
  data: [
    { input: "The cat sat on the mat.", expected: "A cat is on a mat." },
    { input: "The dog barked loudly.",  expected: "A dog is barking." },
  ],
  task: async ({ input }) => {
    const res = await openai.chat.completions.create({
      model: "gpt-4o-mini",
      messages: [
        { role: "system", content: "Summarize in one sentence." },
        { role: "user", content: input },
      ],
    });
    return res.choices[0].message.content!;
  },
  scores: [new Factuality()],
});
 
console.log(result.summary);
// { factuality: { mean: 0.92, p50: 1.0, p95: 0.5, count: 2 } }

Result shape

{
  name: "summarize-quality",
  rows: EvalRow[],         // per-row: input, expected, output, scores, durationMs, error?
  summary: {
    [scorerName]: { mean, p50, p95, count }
  },
  experimentId?: string,    // present if reported to dashboard
}

Each EvalRow exposes the raw output and per-scorer values, so you can inspect the worst rows directly.

Using a managed dataset

You can substitute a managed dataset for the inline data:

const rows = await plr.datasets.fetchRows("faq-v3");
 
await Eval("faq-quality", { data: rows, task, scores: [new Factuality()] });

See Datasets for managing versioned datasets and creating them from existing traces.

CI usage

Eval() returns a structured result; assert in your test runner:

import { test, expect } from "vitest";
 
test("faq agent factuality >= 0.9", async () => {
  const result = await Eval("faq-v3", {
    data,
    task,
    scores: [new Factuality()],
  });
  expect(result.summary.factuality.mean).toBeGreaterThanOrEqual(0.9);
});

Polarity also ships a Vitest helper (@polarityinc/polarity/vitest) that wires scoring into Vitest's reporter — see SDK reference.

Eval vs spec experiments

Use Eval() while iterating on prompts and scorers. Promote to a spec experiment when you need a sandbox or a hard regression gate. The dashboard treats both as first-class runs and the comparison views work across them.