Arize test: Jev's probabilities flag its own errors and hard cases for LLM judges
Arize AI says Jev, a model that returns a probability for each label instead of generating text, can show where an evaluation is likely wrong or hard from a single call. In a test of 36,190 judgments across 10 Phoenix evaluators, Jev's low-probability answers were far more likely to be wrong than its high-probability ones.
- Arize ran 36,190 judgments: seven judges, 10 Phoenix evaluators, 517 labeled examples, ten runs each.
- Jev scored 98.3% accuracy both as a drop-in and natively, averaged across the 10 evaluators.
- Jev changed its answer on 0.97% of examples as a drop-in and 0.19% native, the lowest of any judge.
- Jev's probability detected its own errors with an ROC AUC of 0.95 drop-in and 0.99 native, versus 0.62 to 0.84 for LLMs.
- Jev's entropy predicted LLM instability with an ROC AUC of 0.85, and a 0.5-bit cutoff flagged 11% of examples for review.
How Arize tested the judges
The test was run by an Arize author who has previously argued that an LLM judge's nondeterminism is a signal, not just a problem. The recommended practice is to run a judge ten or more times at high temperature when developing a new eval, because it wavers most on hard examples and ambiguous criteria. According to Arize, that is too expensive at production scale, so the question was whether a judge that returns a probability could give the same signal from one call.
Arize used the human-labeled benchmarks it maintains for each evaluator built into Phoenix: correctness, PII detection, completeness, tool invocation, tool response handling, retrieval relevance, user friction, hallucination, refusal and conciseness. Together they hold 517 labeled examples, between 23 and 136 per evaluator. Seven judges ran every example ten times: three small LLMs (GPT-5 nano, Gemini 3.8 Flash, Claude Haiku 4.5), two frontier LLMs (GPT-5.6 Sol, Claude Opus 5) and two Jev configurations. The whole sweep cost $169.84, of which Jev accounted for 64 cents, according to Arize.
The PII detection suite contains only examples with PII, so it measures how often a judge catches PII, not how often it raises false alarms.
Drop-in versus native Jev
Phoenix's built-in evaluators use prompt templates written for LLMs, with rubric, data and allowed labels in one block of text. Jev's API splits a request into three fields: state for the data to judge, instructions for what to decide, and criteria for the allowed labels with a description of each. Arize ran Jev both as a drop-in, with the full Phoenix prompt in state, and natively, with the rubric split across the three fields. Drop-in is how phoenix-evals runs Jev today.
Averaged across the 10 evaluators, Jev scored 98.3% accuracy in both setups, and none of the per-task differences was significant. Native's biggest gain, 7.1 points, came on completeness, where conversations average around 1,950 tokens. Arize suggests native is worth a try for evaluators that read long inputs.
Consistency and self-diagnosis
A judge was counted as changing its answer when at least one of an example's ten runs disagreed with the rest. Jev did so on 0.97% of examples as a drop-in and 0.19% native, the lowest of any judge. Drop-in Jev was significantly steadier than three of the five LLMs, including GPT-5 nano and Haiku 4.5, and level with Gemini 3.8 Flash and Opus 5. GPT-5 nano changed its answer on 26% of tool invocation examples, and Haiku 4.5 on 12% of PII detection examples.
Jev's probability also tracked correctness. Its median probability on the chosen label was 1.00 for correct answers and 0.71 for wrong ones. All 3,561 runs where Jev put at least 99% on its answer were correct, while below 70% it was right only 73% of the time. As a detector of its own mistakes, the probability scored an ROC AUC of 0.95 as a drop-in and 0.99 native. For the LLMs, which returned labels without per-label probabilities, Arize used agreement across ten calls as the uncertainty signal, and their ROC AUC ranged from 0.62 to 0.84.
Many LLM mistakes are consistent: of examples an LLM got wrong on most runs, 27 to 75% never wavered, so repetition would not catch them. Each judge got only 4 to 14 examples wrong, and Arize says the numbers should be read as a strong direction, not a precise ranking.
Using uncertainty to direct human review
Arize also compared Jev's uncertainty, measured as entropy, with whether any of the five LLM judges changed its answer on the same example. On the 306 examples where Jev was nearly certain, an LLM judge wavered on 3%. On the 24 where Jev was close to a coin flip, the LLMs wavered on 71%. The rank correlation is 0.47 and survives a test that shuffles values within each evaluator. As a detector of examples that make LLM judges unstable, Jev's entropy has an ROC AUC of 0.85.
A 0.5-bit cutoff flags 11% of examples, and the LLMs wavered on nearly two thirds of them. According to Arize, those are the examples a human should review, found without running an LLM ten times.
What it means for eval teams
Per the pricing Arize lists, Jev costs $0.042 per million input tokens with free output, against $0.05 and $0.40 for GPT-5 nano. For teams running LLM judges at scale, the results suggest a single Jev call can both classify and indicate which cases deserve human attention. The benchmarks are small, with 23 to 136 examples per evaluator, so the findings are directional.
Arize AI: What Jev’s probabilities reveal that repeated LLM judgments miss
