Sampling and Backfill

Evaluators cost money on your own provider key, once per judged unit. Two controls decide how much: sampling, which bounds ongoing spend, and backfill, which is the only way to spend on history.

Sampling

The Sampling Rate (%) in step 5 of the editor is the share of matching units an evaluator judges. It defaults to 100. Sampling is decided per unit, using a stable hash of the evaluator and the unit’s own identifier. That gives it four properties worth understanding:

Property	What it means for you
Stable	The same unit always lands in the same bucket, so a restart or a re-run picks the same units rather than a fresh random subset.
Additive	Raising 10% to 20% keeps the original 10% and adds a new 10%. Lowering it drops the top buckets. Verdicts already stored are unaffected either way.
Independent per evaluator	Two evaluators at 10% draw two different tenths. They can overlap, but they no longer land on exactly the same traces, so coverage spreads.
Per unit, not per span	A session evaluator at 10% judges 10% of sessions, and every span of a selected session is still read into the prompt.

A rate of 0 disables the evaluator entirely. It is a pause switch, not “judge a little”, because nothing is selected and nothing is judged.

Sampling is the right lever when an evaluator cannot keep up with your traffic. A judge call takes as long as it takes, so if units arrive faster than they can be judged, the backlog grows without bound and eventually ages out unjudged. Lowering the rate fixes that, and nothing else does.

Sampling applies to continuous sweeps and to backfills, so a backfill reproduces exactly the subset continuous mode would have picked. It does not apply to Test Evaluator, which judges the one unit you selected.

Why you need a backfill at all

Continuous mode is forward-looking by design. An evaluator judges what arrives after you start it, with no lookback, and resuming a paused evaluator restarts from the moment you resumed, so the pause window is skipped too. That is deliberate. Nobody should be silently billed for judging everything already in their account the moment they save an evaluator, so reaching history is an explicit, separate action:

Situation	Is that window judged?
You created and started an evaluator	No, it starts from now.
You paused it for an hour, then resumed	No, so queue a backfill over that hour.
You edited its prompt while it was running	The window in flight is not replayed, and it carries on forward.
Atatus’s evaluation worker was down for an hour	Yes. The evaluator never stopped, so the gap is swept as ordinary work.

Queue a backfill

Use the editor’s One-Time Backfill section in step 5:

Set a start and end time.
Optionally set Max units to judge. Leave it empty for no limit, in which case the time range is the only bound and Atatus asks you to confirm before queueing it.
Queue the job. Progress is shown as it runs.

Five rules govern what you can queue:

Rule	Detail
Independent of Run Continuously	One covers what arrives from now on, the other what is already stored. A draft can be backfilled.
The evaluator must be complete	Project, judge model and prompt are required. Its status is not checked.
One job at a time per evaluator	A second request is rejected, because two concurrent backfills would judge the same units twice and bill for both.
A limit is honored exactly as asked	It is never quietly raised or clamped, because a limit you cannot see is indistinguishable from having judged everything.
The continuous cursor is untouched	A backfill never moves it in either direction.

How a backfill runs

A backfill behaves like a continuous sweep over a range you choose:

It reuses the same selection as a continuous sweep, with the same filter, the same sampling, the same settle rules and the same successful-spans-only rule. A backfill therefore judges exactly the units continuous mode would have.
It walks the range forward in batches, saving progress as it goes, so a cancellation is honored part way and an interrupted job resumes from where it stopped rather than restarting.
Only one backfill runs at a time across the whole system. It is the one path that can spend a lot of money quickly, so it does not get to run several at once.
The judge credential is checked once, up front. A revoked key fails the job immediately instead of walking the whole range and judging nothing.
A job whose window ends at the current time leaves the still-running tail to the live sweep.

A job reports one of five statuses:

Status	Meaning
`pending`	Queued, not started
`running`	In progress, and the count of judged units updates as it goes
`completed`	Finished the range, or reached the limit you set
`cancelled`	You stopped it. Verdicts already written stay.
`failed`	Could not run, most often because of a judge credential problem

Verdicts written by a backfill are recorded as backfilled, so they are distinguishable from continuously judged ones.

Estimating the cost first

There is no cost preview in the editor, so estimate before queueing a wide range:

Set your filter in step 2 and look at the sample table in step 3 to confirm it matches what you expect.
Run Test Evaluator once. The result shows the token counts for a single judgement.
Multiply by the number of units in your range, times your sampling rate.

A backfill over a wide range with no limit and a 100% sampling rate can be a very large number of judge calls. Set Max units to judge the first time you run one for a new evaluator, see what it costs, then decide.

Common uses

Most backfills are one of these four:

Goal	How
Judge the window an evaluator was paused for	Backfill exactly that range.
Re-score data with an improved prompt	Save the new prompt, then backfill the range. The newer verdict supersedes the older one for the same unit.
Add reasoning to verdicts judged before you turned Explanations on	Turn the switch on, then backfill the range.
Try an evaluator on real data before running it live	Create it as a draft and backfill a narrow range with a small limit.

​Sampling

​Why you need a backfill at all

​Queue a backfill

​How a backfill runs

​Estimating the cost first

​Common uses

Sampling

Why you need a backfill at all

Queue a backfill

How a backfill runs

Estimating the cost first

Common uses