This page walks through creating a working evaluator from start to finish. It takes about five minutes, and the only thing you need up front is an API key for the model you want to judge with.

​
1. Add a judge key

Judge calls are made with your own provider credential, so Atatus needs one before an evaluator can run. To add one:
  1. Go to Settings » LLM Keys.
  2. Click New LLM Key.
  3. Pick a provider, paste the API key, choose a model, and give it a name.
A small, fast model is usually the right judge. The task is picking one label from a short list rather than writing prose, so a mini or flash model is both cheaper and faster, and you pay for it once per judged unit.

​
2. Start from a template

Go to LLM » Evaluators and click New Evaluator. You get a catalog of 28 prebuilt evaluators grouped by category, plus Create from scratch if you want to write your own prompt. Pick the one that matches the question you have. If you are tracing coding agents, Task Completion, Step Efficiency and Tool Calling are written for agent traces. If you are tracing your own application, Hallucination, Q&A and RAG Relevancy are the usual starting points. See Evaluator templates for the full list, including which scopes each one supports.

​
3. Set the scope, judge model and prompt

The left side of the editor holds the definition itself:
Changing the scope rewrites the prompt only while it is still the unmodified template you started from. Once you have edited the prompt, Atatus never overwrites your text, so you can switch scope and keep what you wrote.

​
4. Point it at your data

The Configuration column on the right is five numbered steps.

​
Step 1: Pick a Data Source

Choose the project whose spans this evaluator judges. An evaluator belongs to exactly one project.

​
Step 2: Filter Spans

Narrow which spans qualify, using the same filter syntax as the Traces page. Examples are subtype:tool, modelName:claude-opus-5 and harness:claude-code. Leave it empty to consider everything.
The filter decides which units are judged, not which spans of a unit the judge reads. At trace and session scope a trace qualifies if any of its spans match, and it is then judged whole. This is deliberate: a subtype:tool filter that also pruned the judge’s context would leave a task completion evaluator grading a conversation with the user’s question and the model’s answer removed.

​
Step 3: Sample Data

This is a table of real spans, traces or sessions matching your filter, in the time range chosen by the picker in the section header. Select one row, and it becomes the subject of the preview and the test.

​
Step 4: Map Variable to Data

Every {placeholder} in the prompt needs a source. Two kinds are available: Variables the scope supplies show a note instead of a dropdown. Those are tool_calls, tool_call and tool_definitions at trace scope, and conversation and turn_data at session scope. They are assembled for you, and they win over an explicit binding. As you map, a preview underneath renders the prompt with the real data from your selected sample row. The preview is free, because it makes no judge call.

​
Step 5: Task Configuration

This step holds the three controls that decide how much data gets judged:

​
5. Test it on one real unit

Click Test Evaluator. This runs the whole pipeline once against the row you selected in step 3, and shows the verdict inline. Three things are worth knowing about a test:
  • It costs one judge call on your key.
  • It stores nothing, so a test never becomes a chip.
  • It ignores sampling and always tests what is on screen, including unsaved edits.
Two failure shapes are reported differently, because the fix is different:
The test path and the real run use slightly different mechanisms to hold the judge to your choices, so a model that behaves perfectly in a test can be marginally worse in production. Any prompt ending with an explicit “answer with exactly one of these words” instruction is safe under both.

​
6. Save and start it

Three buttons in the footer do three different things: Two conditions must both hold before anything is judged. The evaluator has to be active, and Run Continuously has to be on.
A new evaluator opens with Run Continuously already on, so Create and run evaluator is all you need. The two switches are still independent: Create and run sets the status to active but does not touch Run Continuously, so if you turned that switch off yourself, the evaluator saves as active and judges nothing. The evaluators list badges that state as paused, so check the badge after saving.

​
7. Wait for the first chips

Definition changes reach the worker within 30 seconds, and a new credential within 60 seconds. After that, the wait is the scope’s settle delay plus one sweep: The very first evaluator on an account can take up to another minute before the chip column appears on the lists at all.
The first sweep is always empty. An evaluator starts judging from the moment you start it, so there is nothing yet in its first window. Traffic from before that point is only reachable with a backfill.

​
Next steps

Once chips are appearing, these pages cover what to do with them: