Let Evals catch the change before it reaches your customers

Evals let you test a change before it reaches a customer. Replay real past runs against a candidate revision, and let a judge model score each result as pass or fail. Shared across Tasks and agents, and built to catch a regression before it becomes a support ticket.

That's the summary. The rest of this is why it's built the way it is, because the design decisions are the interesting part.

The demo ends, the release doesn't

Say you run developer support for an API platform. An agent monitors incoming issue reports, opens a tracker for each, and writes an escalation brief for whoever picks it up next. It works, quietly, for weeks.

Then customer success asks for one more thing: every triage should end with an action plan the team can send straight back to the customer.

That reads like a copy edit. It isn't one.

The new version has to keep the original summary, evidence, impact, and escalation notes exactly as they were, and add severity, an owner, a next action and a customer-facing reply on top. Once a team depends on what an agent produces, an instruction edit stops being text and starts being a release.

Just eyeball it

The first attempt at this is always the same. Change the instruction, run the new version by hand against a couple of inputs you remember being tricky, read the output, and decide it looks fine.

Works fine, right up until the input you didn't happen to try is the one a customer sends.

Manual review has no memory. The edge case that broke things three revisions ago isn't the one anyone thinks to check today, because nobody wrote it down anywhere the next reviewer would look. Every check starts from a blank page, and every blank page misses something the last one caught.

Test the regression, not the intuition

So the test runs against real cases instead of memory. An eval set is a reusable collection of them, scoped to one Task or agent, built from task_run_ids you already have. Not fixtures somebody wrote up front and stopped maintaining. Actual runs your production traffic already generated.

In practice, this means running the current version against a handful of real inputs it has already handled, and keeping those runs as the reference: the behaviour the new version isn't allowed to lose.

Worth noting while we're here: an eval set is the same shape whether it sits under a Task or an agent, created, listed, updated and deleted like any other resource over the API.

Build the set once. Every future revision gets checked against the same bar the last one cleared, not against whatever a reviewer happens to remember.

What a judged run looks like

Comparing outputs byte for byte doesn't work for anything a model writes in free text, and almost everything worth checking is free text. Two correct answers can be phrased completely differently, and a diff would flag both as regressions.

So the comparison runs through a judge model instead. It reads the candidate's output next to the reference run it's being checked against, and returns a verdict, pass or fail, with reasoning attached. The judge runs on a different provider than the model being evaluated, because a model grading its own output has an obvious incentive to agree with itself.

The triage candidate kept every original section and added a graded severity, a named likely owner, a reproduction plan for the next day, and a draft reply ready to send. More output, same foundation, checked by something other than the person who wrote the prompt.

Pass isn't the only outcome worth naming

A clean run means no regressions and no errors, which took a while to land as two separate things worth distinguishing.

A failure means the judge compared two outputs and found a real regression: the candidate got something wrong that the reference got right. An error means the case couldn't be evaluated cleanly at all, a broken run, a missing output, something upstream of the judge entirely. Different problems, different fixes. Burying both under one red mark tells you something is wrong without telling you which kind.

The part that actually shaped the ship decision here wasn't the pass at all. Token cost can barely move between reference and candidate while duration moves a lot, because a richer output takes more work to produce. That tradeoff, more useful and measurably slower, is exactly what an eval exists to surface before a customer feels it rather than after.

The part release owners care about

Each case comes back with a verdict and the reasoning behind it. Across the whole set, that adds up to how many passed and how many failed, alongside what changed in cost, credits and speed between the old version and the new one. All of it rolls into a single pass rate, the number built for a pipeline to check rather than a person reading case by case.

Name your own reference runs for a formal gate, or send the candidate on its own for a quick check, and Rightbrain samples the most recent usable runs itself, skipping anything an earlier eval already generated. If there aren't enough runs to sample from, it returns an error rather than quietly grading a thinner set and calling it done.

Set a minimum pass rate, run the eval against a candidate revision before promoting it, and the revision only goes live once it clears the bar. The eval result and the revision it tested stay attached to each other, the same way approval policy stays attached to the revision it governs.

"Did this change break anything" gets an actual answer, not a memory of what looked fine last time.

Where the gate actually sits

Product teams add the capability without wondering what they broke further back. Customer success compares before and after and confirms the new answer is genuinely more useful, not just longer. Whoever owns the credit spend sees the cost of a revision before it reaches a single customer.

Which is roughly the difference between an agent someone edited and hoped, and an agent someone edited and checked. The agent still does the extra work the business asked for. The evidence just stops being a guess, and starts being something you can point to.

Full endpoint reference and configuration detail: Evals.