Ollama adds decision models for fast, typed answers on local machines

How the new API works

According to Ollama, the new API is meant for tasks that need fast, structured decisions, such as ticket triage, model routing, and content or safety moderation. A developer sends text as state together with a set of named questions, and the model answers all of them in one request.

The questions come in several types. In the example published by Ollama, a customer support ticket reads "I was charged twice. Please refund the extra payment." Three questions are attached to it: which team should handle the ticket (a choice question with the options billing, technical and other), whether the customer explicitly asks for a refund (a noul question), and how urgent the ticket is (a score question with the levels Routine, Soon and Urgent).

The response is typed. For the team question, the model returned the choice "billing" with probabilities of 0.985 for billing, 0.012 for technical and 0.003 for other, plus a confidence value of 0.922. The refund question returned a noul value of 0.997, and the urgency question returned a score of 0.815. The usage field in the example reported 841 input tokens and 4 output tokens.

Speed and available models

Ollama says the main advantages are no additional costs and lower latency when models run locally, since requests do not have to travel over a network. As an example, the company reports that Nimble 9B averaged 91ms per decision in a Pac-Man demonstration running on an M5 Max MacBook Pro. Ollama describes that as fast enough for rapid decisions such as playing a game or processing content in real time.

Three decision models are available today:

• nimble: an open-source 9B parameter decision model developed by Bespoke Labs
• tev1: an experimental 4B decision model from Together AI
• tev1:0.8b: an experimental 0.8B decision model from Together AI

Ollama also says more decision models are coming soon, including models served by Ollama's cloud.

The company published a benchmark chart based on Bespoke Labs public benchmarks, measuring mean accuracy across 13 public data sets with human labels, covering 3,880 decisions. Nimble and Tev1 were evaluated on Ollama, while the Jev 1.13 figure comes from Bespoke Labs' published run on the same decisions. Ollama links to its evaluation results and benchmark suite for details.

Getting started and what comes next

Users need to download or upgrade to the latest version of Ollama and then pull a model, for example with ollama pull nimble. Requests can be sent with curl to the local /v1/systemone endpoint or through TypeSafe's official Python SDK, which is installed as typesafe-sdk. The SDK is configured with environment variables for the base URL (http://localhost:11434), an API key and a default model.

In the Python example, developers define questions with the Choice, Noul and Score classes and call the client's system_one method. Results are then read from typed fields such as the chosen team, the refund value and the urgency score.

Ollama calls this the first of many releases adding decision model support. Planned updates include faster performance on Apple Silicon powered by MLX, and more models specializing in different kinds of decision making.

For developers, the release offers a way to run classification and routing steps locally with structured outputs, without per-request costs, according to Ollama. How the models compare in accuracy is set out in the company's benchmark materials.