Arize benchmarks eight decision models: Jev, Liquid d1 and Clef 27B close on accuracy

What decision models are and why Arize tested them

According to Arize, a decision model takes a question and a fixed set of options and returns a probability for each option. It does not write text, so there are no output tokens to pay for or wait on. Arize says this makes such models cheap and fast for two jobs: making a single decision inside an agent, such as which tool or skill to use or whether to stop, and acting as the judge that scores traces in Arize AX.

The category took off quickly. TypeSafe released Jev on 15 September, and within a few weeks about 10 companies had shipped something that works the same way, according to Arize. OpenAI, Liquid, Cloudflare, AWS and PostHog all launched one, and an open clone called Kev appeared on Hugging Face. Arize says seven of the eight models it tested accept Jev's /v1/systemone request format. The exception is OpenAI, whose Decisions API takes one block of plain text plus a list of questions. Arize wrote each case as labelled text sections for it, and cautions that a different layout could score differently. It describes OpenAI's numbers as a first look at a public beta.

The contestants and the rules

The eight models are Jev 1.13 (TypeSafe), Liquid d1 (Liquid AI), OpenAI Decisions (gpt-6-luna), Clef 27B (Cloudflare), Kev 9B (Jared Palmer), Jeeves 9B (PostHog), Strands Decider 2B (AWS) and Laya multilingual (Convai Innovations). Laya is a 322M-parameter BERT-style encoder rather than an LLM. List prices per million input tokens were $0.042 for Jev, $0.04 for Liquid d1, $0.10 for OpenAI and $0.24 for Clef 27B. Kev, Jeeves, Strands and Laya are free to self-host.

Arize split the field into a cloud conference of hosted APIs and a local conference of open models run on a MacBook Pro with an Apple M4 Max and 36 GB of memory. Each conference crowns a champion, and the two meet in a grand final. The judges score accuracy, latency and cost per 1,000 decisions, and a model needs two wins out of three to advance. Accuracy counts as a win only when 95% confidence intervals do not overlap. Each model gets its own yes/no cutoff, tuned on one half of the data and scored on the other.

The three challenges were hallucination detection on 2,675 rows from RAGTruth, all 13 Phoenix evaluator suites with 655 labelled cases, and 52 agent routing decisions.

Quarterfinal results on hallucination detection

Jev beat OpenAI Decisions. Jev scored 85.5% balanced accuracy to OpenAI's 81.5%, but the confidence intervals touch, so accuracy was a draw. Jev answered in 0.08 seconds per decision against 0.16, and cost $0.045 per 1,000 decisions against $0.090, so it advanced 2-0.

Liquid d1 beat Clef 27B. Accuracy was effectively level, at 85.2% for Clef 27B and 84.8% for Liquid. Liquid was faster, at 0.20 seconds against 0.50, and cheaper, at $0.034 per 1,000 decisions against $0.223.

In the local bracket, Kev beat Laya. Laya was the fastest model in the competition at 0.04 seconds per decision and is under a gigabyte, but it scored 59.8% to Kev's 76.8%. Jeeves beat Strands, 80.3% to 69.0%, because it reasons before answering. Strands was about 50 times faster at the median.

Arize also compared the survivors with an LLM judge. In its earlier benchmarks, Opus 5 scored 86.6% balanced accuracy at $14.30 per 1,000 judgments. Jev, Liquid and Clef 27B land within the noise of that score, according to Arize, at 0.3%, 0.2% and 1.6% of Opus 5's cost respectively. OpenAI sits a few points further back on accuracy, at 0.6% of the cost.

Conference finals: Jev and Liquid draw on the Phoenix evals

The second round tested whether a model could serve as the judge in an eval pipeline. Here Arize used calibration as the tiebreaker, measured as the average gap between a model's stated confidence and how often it is right, where lower is better.

In the cloud final, Jev scored 91.1% and Liquid 87.6%, with overlapping intervals, so accuracy was a draw. Jev won on speed and Liquid on cost. Jev's calibration error was 0.023 against 0.051 for Liquid, but with 655 cases the intervals overlap and Arize ruled the match a draw. If forced to pick, Arize said, Jev edges it. Given one correct and one incorrect case, Jev was more confident about the correct one 85% of the time, against 81% for Liquid. On the 14% of cases where Jev said it was at least 99% sure, it was right every time.

The source text is cut off before the local final, the grand final on agent routing and the closing results.

What it means for agent builders

For teams building agents or running evals, the benchmark suggests the choice between decision models will rest less on accuracy than on latency, cost, where the model runs and how sensitive it is to question wording, based on Arize's finding that the leaders were closely matched on accuracy. The cost gap against a full LLM judge is large in Arize's figures, though the results come from Arize's own tests and, for OpenAI, from a public beta model run with a prompt layout Arize chose.