Anaconda Tests TypeSafe's Jev Decision Model on Real Customer Complaints
Anaconda has published an analysis of Jev, a "decision model" released by TypeSafe that reads an input once and returns a probability for every answer instead of writing text. The post describes what is known about the model and compares it with a small open classifier Anaconda trained itself.
- TypeSafe's Jev is a decision model that returns a probability for every answer instead of generating text.
- According to Anaconda, Jev costs $0.042 per million input tokens, with a median of 88 milliseconds per answer.
- Reordering options changed Jev's probabilities by up to 0.20 and crossed 0.9 in 7 of 80 cases.
- On new math problems Jev was right 56% of the time while giving itself about 35%.
- Anaconda advises testing probabilities and option order on your own data before deployment.
What a decision model does
Anaconda frames the problem with a customer complaint: a person is billed twice for a $340 hotel booking, the second charge ends up with a collections agency and appears on a credit report. Such a complaint needs a resolution path, a yes or no on whether a person should look at it quickly, and a severity grade. Most organizations, Anaconda says, combine hand-written rules, a conventional machine learning model and a human fallback. A growing fourth option is to hand the complaint to a large language model and ask for JSON.
Anaconda calls that an odd way to get a one-word answer. The model writes token by token, may reason through hundreds of words that are billed, and needs a schema, a validator and retry code. Its stated confidence, such as "confidence": 0.91, is text it chose to write, not a measured probability.
TypeSafe calls Jev a "System One model", after Daniel Kahneman's Thinking, Fast and Slow. It is a classifier: given an input (the state) and a set of questions with fixed answer lists, it returns a probability for every answer. It cannot summarize, write code or explain its choice. According to Anaconda, OpenAI has since announced a similar Decisions API, in limited preview, at DevDay on 29 September.
Why developers care, and what is known about Jev
Sebastian Raschka, who has long built text classifiers, found Jev worked better than he expected. He calls it "the ChatGPT moment for classification". Anaconda lists three advantages:
- The answer is always one of the supplied options, so only correctness needs checking.
- The probability is read from the model's internal state, not written by it, so it can be checked against outcomes and thresholded.
- It is cheap: $0.042 per million input tokens and nothing for output.
Anaconda's complaints averaged about 630 tokens, costing around $0.00003 each, with a median of 88 milliseconds per answer. Raschka ran the 25,000 IMDb test reviews for about 65 cents and scored 96.5%, on par with a fine-tuned ModernBERT model. Anaconda warns that the API's confidence field is not an accuracy estimate but a measure of how far the top answer stands out, so thresholds should be set on the probability.
TypeSafe has not published how Jev is built, and the model cannot be downloaded. Anaconda follows Archer Hume's analysis of about ten thousand probing API calls. TypeSafe says all answers are produced at once and that the model is an existing language model trained with an unpublished method called Reinforcement Learning for Calibrated Decisions (RLCD). Its CEO has said all training data is synthetic. API tests show questions in the same request cannot see each other, and that adding an irrelevant fifth option shifted the balance between two original options. Hume infers a large general language model underneath, while Raschka guesses something small like ModernBERT.
Where the promise falls short
Anaconda stresses that good benchmark probabilities do not guarantee good probabilities on your own data. On freshly written math problems, Jev was right 56% of the time while giving itself about 35%, which Anaconda calls far too cautious.
Option order also matters. On 40 complaints, Jev picked the same answer after reordering in 79 of 80 cases, but its probabilities moved by up to 0.20 and crossed 0.9 in 7 of the 80. Some of this is noise: an untouched question moved by up to 0.04 between identical requests. Anaconda's own model kept its answer in about 48 of 80 cases.
Jev also cannot reason in steps or look things up. Anaconda endorses TypeSafe's suggestion that the decision model decides and a language model writes.
What it means for users
At Jev's price, Anaconda argues, you can decide on every row, event or log line. The practical advice from the post is to measure probabilities on examples from your own workflow where the answer is known, and to test option order before deploying any such model, since a shift from 0.84 to 0.95 can cross a threshold and change what a system does.
Anaconda: Decision Models: Jev vs. a Fine-Tuned Classifier
