Every evaluator starts from a template. A template is a prompt, a set of labels, and a declaration of which variables it reads. All three are editable once you have created the evaluator from it. The tables below list every prebuilt template. The Reads column is the set of variables you have to map, minus anything the scope supplies for free.
Direction is how a higher score reads, and it drives the chip colors and the ordering. On a minimize evaluator the first label is the bad outcome and renders red. On a maximize evaluator the first label is the good one.

​
Correctness and quality

These six judge whether an answer is right, grounded, or faithful to its source:

​
Code evaluation

These three judge generated code:

​
Agent and trace evaluation

These are the templates written for agent runs, and the ones to reach for with coding agent traces. They read the trace’s tool calls, which no single span has, so they have no span-scope variant:
The variables listed after “plus” are supplied by trace scope itself. tool_calls arrives as an ordered JSON array of {name, arguments, result}, and tool_definitions as the distinct tools that trace used. You do not map them, and a trace that called nothing renders [].

​
Conversation and sentiment

One template reads the user’s own words rather than the model’s:

​
Session evaluation

Session templates read the whole conversation, so they exist at session scope only and need no mapping at all. The conversation variable is assembled for you:

​
Content safety and moderation

These seven check what your model returns, with one exception that checks what your users send:
Toxicity and Toxicity Check are not duplicates. One judges what your users send you, and the other judges what your model sends back. Run both if you care about both directions.

​
Security

All four read the incoming prompt, so they are the ones to run against user-facing input:

​
Scopes, and why a template does not offer all three

Changing an evaluator’s scope changes the material the judge is shown, so a template only offers the scopes where its criterion still makes sense. When you switch scope, only the data section of the prompt is rewritten, plus one line saying what the material now is. The instructions are never touched, because the criterion does not change with granularity. Six templates have no session variant, because they need a reference that the conversation does not carry, such as a retrieval context, an expert answer, or a tool catalog. Those six are Hallucination, Q&A, Human vs AI, RAG Relevancy, Reference Link Correctness and Tool Calling. A template with no variant for the scope you picked tells you which scopes it is written for.

​
Create from scratch

Create from scratch gives you an empty prompt, two placeholder labels and no direction. Use it when none of the prebuilt templates fits. The rules are the same as any template:
  • You need two or more choices, each with a score.
  • Every {placeholder} in your prompt must be mapped, unless the scope supplies it.
  • End the prompt with an explicit instruction to answer with exactly one of your labels. The judge is already constrained to your choices, but a prompt that also states the rail in words is markedly more reliable, and every prebuilt template does this.
The fastest way to write a custom evaluator is to create one from the closest prebuilt template and edit its prompt. You inherit a working set of choices, a mapped set of variables and a proven prompt shape, and Atatus will not overwrite your edits when you change scope.