Evaluators Overview

An evaluator scores your LLM data using another LLM. You give it a prompt and a fixed set of choices, Atatus asks a judge model to pick one of those choices for each span, trace or session, and the answer is stored as a verdict and shown as a colored chip next to the row it judged. Evaluators answer the questions a dashboard cannot. Latency and cost are measurements, but whether an answer was grounded, whether an agent actually finished the task, and whether a prompt tried to jailbreak your system are all judgements.

​

What you need before you start

There are two prerequisites:

​

The three parts of an evaluator

Every evaluator is built from three things: Everything else, such as the filter, the sampling rate and whether it runs continuously, controls which of your data gets judged rather than what the judgement is.

​

Scope decides what the judge sees

Scope is the most important choice you make, because it decides both the material the judge reads and which variables your prompt is allowed to use: A variable the scope supplies is assembled for you and needs no mapping. At trace scope, tool_calls is built from the trace’s tool spans as an ordered JSON array of {name, arguments, result}, and a trace that called no tools renders []. At session scope, conversation is the whole back and forth.

​

How a verdict gets made

A verdict goes through five steps:

1. Atatus looks for units, meaning spans, traces or sessions, that arrived since the evaluator was last swept and have settled.
2. Your filter narrows those units, then your sampling rate picks a stable subset of what is left.
3. For each selected unit, the variables are resolved from the real span data and your prompt is rendered.
4. The judge model is called once, and it is constrained to answer with one of your choices.
5. The chosen label and its score are stored as a verdict, and appear as a chip.

Sweeps run about once a minute, and a unit only becomes eligible once it has been quiet long enough to count as finished: In practice, expect a chip roughly 1 to 2 minutes after a span at span scope, 2 to 3 minutes after the last span of a trace, and about 30 minutes after a session goes quiet. Coding agent traces get one extra check. A trace whose root span has not arrived yet is still in progress, so it is not judged, and its place in the queue is held for up to 30 minutes rather than skipped. Traces from your own instrumented application are exempt, because their root is an upstream request that the LLM spans never contain.

​

Only successful spans are judged

Every unit selection and every span read is narrowed to spans that did not error. An errored span never produced the answer there is something to score, so judging it would grade the outage and bill you for the verdict. At trace and session scope this applies inside the unit, so a partly errored trace is still judged on its successful spans only, and a unit with no successful span at all disappears.

​

Continuous mode only looks forward

Starting an evaluator means “judge what arrives from now on”. There is no lookback of any kind, which has three consequences:

• A brand new evaluator judges nothing on its first sweep, then tracks live traffic.
• Pausing and resuming an evaluator skips the pause window entirely, because resuming restarts from the moment you resumed rather than from where it stopped.
• An ordinary edit to the prompt, choices, filter or sampling rate does not rewind anything. Units already judged keep their verdict, and new units get the new prompt.

To judge data that is already stored, queue a backfill. That is the one path that knowingly spends money on history, which is why it is never automatic.

​

What it costs

Judge calls bill to your own provider key. Two things bound the spend: One evaluator claims at most 500 units per sweep, so a traffic spike in one project cannot run away with your whole budget in a minute. Prompt and completion tokens are stored on every verdict, so the real cost is measurable after the fact.

​

Limits worth knowing

These limits keep a single prompt and a single sweep bounded:

​

Next steps

Continue with the page that matches what you want to do:

• Create your first evaluator walks through the whole flow.
• Evaluator templates lists the 28 prebuilt evaluators and what each one reads.
• Reading verdicts covers chips, explanations, and filtering your traces by verdict.
• Sampling and backfill covers judging history and controlling spend.
• Troubleshooting is the place to start if nothing is being judged.