Fiddler tests TypeSafe's Jev as a fast, cheap judge for coding agent risk
Fiddler AI tested TypeSafe AI's new Jev model as a judge of coding agent risk and found it returned ratings faster and at a lower estimated cost than two general-purpose LLM judges. The trade-off, according to Fiddler, is that Jev gives probabilities but no written rationale.
- In Fiddler's test of 31 Bash calls, Jev had a median response of 237 ms, versus 898 ms for Gemini 3.5 Flash-Lite and 1,106 ms for GPT-5.4 nano.
- Jev's estimated cost was about $0.0016, against $0.01261760 for Gemini and $0.00813785 for GPT-5.4 nano.
- The three judges disagreed on 11 of 31 commands, and Fiddler had no ground truth to say which was right.
- Jev gives probabilities but no written rationale, which can make decisions harder to investigate.
- Fiddler advises human review, labeled reference cases and a full retest when evaluators or models change.
What Jev is and how it differs from an LLM
TypeSafe AI released Jev in early access the week before Fiddler's post. Fiddler describes it as TypeSafe's first "System One Model", a class of models that make fast, structured decisions that applications can act on directly.
Jev answers in three typed formats. Choice selects an option from a fixed list, Score rates something on an ordered scale, and Noul gives a probability from 0 to 1 that there is enough information to act. Jev evaluates several questions independently and in parallel. An LLM can also return structured JSON, but it generates its response token by token and can add a written rationale. Jev does not. According to Fiddler, TypeSafe's published pricing charges for Jev's input tokens but not its output tokens.
Because the answers are independent, an application has to combine them into a routing rule itself. Jev's probabilities show how it weighs the available answers, not which details of the input drove its choice.
How Fiddler ran the test
Fiddler gave a coding agent twelve tasks in disposable sandboxes and recorded the commands it chose. The team selected 31 Bash calls and paired each command with its recorded outcome. Three judges rated each item for risk, from low to critical, using the same rubric and instructions.
The judges were Jev, google/gemini-3.5-flash-lite and openai/gpt-5.4-nano. Fiddler chose the two general-purpose models because they are described as the lowest-cost tier suitable for classification at the time of writing. Both were set to their lowest reasoning levels so that output-token use would be more comparable. No temperature, top_p or seed was set for the LLM judges.
Fiddler stresses that it had no independently established correct ratings. Agreement shows where decisions align, not which judge is right.
Speed, ratings and cost
Each judge rated the same 31 calls once. Median response times were 237 ms for Jev, 898 ms for Gemini 3.5 Flash-Lite and 1,106 ms for GPT-5.4 nano. Fiddler notes that the LLMs also wrote a rationale, so this is not a like-for-like comparison of rating-only responses.
- Jev: 18 low, 7 medium, 6 high, 0 critical
- Gemini 3.5 Flash-Lite: 22 low, 4 medium, 5 high, 0 critical
- GPT-5.4 nano: 14 low, 10 medium, 7 high, 0 critical
All three judges agreed on 20 commands. Two agreed and one differed on 8, and all three differed on 3, so the judges disagreed on 11 of 31 commands.
On cost, Fiddler estimated Jev at about $0.0016 for the 31 calls, based on 38,315 input tokens at $0.042 per million with output free. Gemini's provider-reported cost was $0.01261760 and GPT-5.4 nano's was $0.00813785. Fiddler puts Jev at roughly 7.8 times lower than Gemini and 5.1 times lower than GPT-5.4 nano. No paid Jev bill was observed, because a Vercel Gateway promotion made the calls free. Prices may change, and the estimate is not a like-for-like measure of value.
Limits and what to do with the results
The missing rationale is the main drawback. When a Jev rating is surprising or consequential, a reviewer may need to go back to the original evidence. Fiddler says written rationales give reviewers concrete claims to check against the full trace. Jev can also only choose from the options defined for it, so Fiddler suggests including "needs review" or "none of the above" when a case may not fit. It also points to TypeSafe's Jev 1.13 guidance, which warns that the model may read questions literally or be distracted by irrelevant details.
On consistency, Fiddler reports that Jev's ratings stayed the same across repeated runs, while the generative judges showed some variation.
For organizations rolling out coding agents, Fiddler's advice is to keep humans in the loop. Reviewers decide which ratings match the organization's risk policy and build labeled cases for testing evaluators. A reference dataset labeled under a specific policy would let teams compare each evaluator with the judgment they expect, and refine instructions and review thresholds where the two differ. Fiddler also recommends retesting the full evaluation workflow whenever an evaluator's configuration changes or a newer model is adopted, rather than assuming earlier results still apply.
The test covers one small set of 31 commands and only the Choice feature. Fiddler says it did not test Jev's Score and Noul types or multiple focused questions.
Fiddler AI: Testing Jev for Coding Agent Evals: Comparing Speed, Cost, and Rationale
