Jev model matches gpt-4o-mini judge on 897 of 1,000 turns, at lower cost

What Jev is and how it differs from LLM-as-a-Judge

Teams running an LLM in production usually grade traffic with an LLM-as-a-Judge setup: each exchange goes to a frontier model, and a verdict is parsed from the response. Comet describes two problems with this. Frontier-model prices push most teams to sample their traffic, so rare failures can sit in the unscored share. And few teams check whether the judge itself is right before trusting its dashboard.

Jev, from TypeSafe AI, targets the first problem. It answers typed questions about a piece of text rather than writing about it. Given a user message and a reply, and asked whether the reply answered the question, it returns a probability instead of a paragraph. According to Comet, that means nothing to parse, no schema to enforce and no temperature or seed to pin down. It bills only the tokens it reads, at $0.042 per million.

Comet argues the approach follows a familiar pattern: embeddings moved to small dedicated encoders, reranking to cross-encoders, and moderation to classifiers like Llama Guard. Jev is the first model of this kind that Opik, Comet's open-source LLM observability and evaluation platform, supports directly.

The test on Ollie

Comet ran the comparison on Ollie, an AI assistant inside Opik that reads and analyzes traces, diagnoses issues and suggests fixes. Ollie already had an LLM-J rule on openai/gpt-4o-mini scoring whether a reply answered the user. The team sampled 1,000 real turns from 578 threads in its production monitoring project and re-logged them into a dedicated project, so both judges scored the same turns without touching live traces. Both ran as online rules at 100% sampling, with one Boolean score and the same question: "The answer addresses the question that was asked."

• Agreement with the other judge: 89.7% for both gpt-4o-mini and Jev
• Cost per 1,000 traces: $0.0993 for gpt-4o-mini, $0.0285 for Jev
• p50 latency: 958 ms for gpt-4o-mini, 253 ms for Jev
• p90 latency: 1,202 ms for gpt-4o-mini, 352 ms for Jev

Jev was cheaper and faster at the median despite sending 1.5× more input tokens, because the Decisions API encodes the question structure alongside the text, Comet says. At that price, the company writes, judging 100% of traffic becomes the default rather than something to argue for.

Where the judges disagreed

The 103 turns on which the judges split clustered where Jev itself was unsure. Of those, 64.1% fell between 0.3 and 0.7 on Jev's probability, against 16.7% of the agreements. Sorting turns by Jev's confidence produced a U-shaped pattern: below 0.2 the judges agreed 100% of the time (n=118), above 0.8 they agreed 95.6% (n=568), and in the 0.4 to 0.6 band agreement fell to 63.5% (n=115).

Jev's outputs were confident but not binary: 568 of 1,000 above 0.8, 115 in the 0.4 to 0.6 band, spread across 97 distinct values. Comet suggests the middle band can serve as a review queue routed to a human, rather than sampling blindly. A judge that returns only true or false cannot distinguish a confident call from a coin flip.

Comet is explicit about a limit: it did not establish which judge was right. That requires labelled data, and labelling raised the question of where Ollie's scope ends. The shared question already scored a polite refusal as a failure. Before evaluating a judge, the company says, someone has to write down the policy it is judging against.

Using Jev inside Opik

Comet describes three integration points. First, a native tracing integration, since the typesafe-sdk client is not OpenAI-compatible: each system_one call becomes an LLM span with state and questions as input, answers as output, plus token usage, provider and cost.

Second, Jev can be selected as the model behind an online LLM-as-a-Judge rule under the OpenRouter provider, using the workspace's existing key. The model is typesafe/jev-1.13, and the evaluation span records the served version, typesafe/jev-1.13-20260917. The prompt holds only the data (INPUT and OUTPUT), while each score's description is the question. Only Boolean scores are supported. A probability of 0.5 or more stores a 1, and the probability is kept in the reason so the threshold can be revisited later.

Third, because Jev has no temperature and no few-shot examples, the wording of the question is its whole configuration surface. Opik's Agent Optimizer can iterate on that wording against a labelled dataset, using agreement with the label as the metric.

What it means for teams

Comet says Jev fits where a number is the whole product. It returns no written critique, tool calls, non-Boolean scores or thread-level judgments. If a human reads the reason, the company advises keeping a generative judge; if the score feeds a dashboard, a filter or a route, a probability is enough.

Comet frames the broader takeaway not as "adopt Jev" but as a repeatable method: test whether a new model does your job better on your own traffic, with both judges seeing the same traces and cost and latency recorded beside the verdicts. If a model lands close but not quite, the company suggests sharpening the question, and if that falls short, using the labelled data to train your own.