Almost every “my evaluator is not working” report is one of the seven conditions below, so start here.

​
The checklist

An evaluator judges new data only when all of these are true. If any one is missing it is skipped silently, and there is no error to find:
A new evaluator opens with Run Continuously already on, so condition 2 normally takes care of itself. It still bites when someone turned the switch off to stop the spend and then forgot, because the two switches are independent: Create and run sets the status to active without touching Run Continuously. The list badges that state as paused, so if the badge says paused while the status says active, the switch is off.

​
Nothing has been judged yet

If the checklist passes, the next question is whether enough time has passed. Evaluators judge finished work, so there is always a delay: Add up to 30 seconds for a definition change to reach the worker, up to 60 seconds for a new judge credential, and up to another minute for the chip column to appear the very first time an account gets an evaluator. Two more things are worth ruling out:
  • The first sweep is always empty, because an evaluator starts from the moment you started it, so its first window contains nothing yet.
  • Your filter may match nothing. Check the sample table in step 3, and if it is empty for a recent time range then the filter is the problem rather than the evaluator.

​
It judged some units but not others

That is usually correct behavior. Four rules narrow what gets judged:

​
Chips stopped appearing

Verdicts are stored only for clean successes, so a judge problem shows up as the chips ceasing. There is no error chip. In order of likelihood:
  1. The judge credential or model is wrong, because of a rotated or revoked API key, or a model your account cannot access. Verify with Test Evaluator, which reports a judge failure directly.
  2. The judge is answering off-rail, returning something outside your choices, so every result is discarded. Test again, since an off-rail result is reported as such. The fix is the prompt: state the exact allowed answers in the text, as every prebuilt template does.
  3. The evaluator was paused. Check the badge.
  4. Your traffic changed. A filter that matched last week’s spans may match none of this week’s.
A failed judgement is not retried. Once you have fixed the credential or the prompt, the units that failed in the meantime need a backfill, because continuous mode has already moved past them.

​
Test Evaluator works but no chips appear

Test and continuous running are independent. A successful test proves the prompt, the mapping and the credential are all good, but it says nothing about whether the evaluator is running. Go back to the checklist, and check the status and the Run Continuously switch. A test also never becomes a chip, because it stores nothing by design.

​
Explanations are not showing

Work through these three in order:
  1. Check the switch is on for the saved evaluator, not only in an unsaved draft.
  2. Check you are looking at verdicts judged after you turned it on. Storing the reasoning is not retroactive, so older chips have no tooltip and never will unless you re-judge them with a backfill.
  3. On the lists, reasoning is a hover tooltip on the chip. For the full text, open the trace, span or session detail panel.

​
Frequently asked

Why was yesterday’s traffic not judged when I started the evaluator? By design. Continuous mode has no lookback at all, so it judges what arrives from the moment you start it. Use a backfill for history. I paused an evaluator for an hour and resumed it. Is that hour judged? No. Resuming restarts from the moment you resumed, so backfill that hour if you want it. Does Test Evaluator cost money? Yes, one judge call on your provider key. The live preview of the rendered prompt under the mapping rows costs nothing. Can I edit an active evaluator’s prompt without re-judging everything? Yes. An edit does not rewind anything, so units already judged keep their verdict and new units get the new prompt. Can two evaluators judge the same trace? Yes, and you get one chip per evaluator. Sampling is independent per evaluator, so two evaluators at 10% draw different tenths, but nothing prevents an overlap. What happens to verdicts if I delete the evaluator? They stay, and the chips keep rendering until they age out with the retention of the spans they judged. Why does a chip show a count next to the label? Because that many spans of the trace got the same label from that evaluator. See Reading verdicts. Why is a session’s chip on only one row? A session verdict is anchored to the session’s first trace, which is the only trace-grained row it can render on. Can I evaluate non-LLM spans? No. Evaluators only see LLM spans, because ordinary APM spans carry none of the model fields, so the variables would resolve empty. Why does the sample table show more rows than my sampling rate suggests? The sample table shows what your filter matches, so you can see the data you are about to judge. Sampling is applied when units are actually selected for judging. What time zone are the times in? UTC throughout. Can I judge a range longer than my data retention? No. The date picker will let you ask for it, and the request is rejected rather than silently returning less.