Arize Tests Confidence Gate for AI Agent Tool Selection in Phoenix

Splitting the choice from the call

Most agents select tools in a single model call: the chat model reads the conversation, picks a tool and writes its arguments. According to Arize, the trace then shows the call, its arguments and its result, but not the choice itself. You cannot see which other tools the model considered or how close the decision was.

Arize tried a different split. A decision model, in this case OpenAI's Decisions API, takes a list of options and returns a probability for each one without writing any text. The chat model keeps the second job: calling the chosen tool and filling in the arguments. Phoenix now supports decision tracing with a new span kind called DECISION, and each pick lands on its own span, with a probability for every option the model rejected.

The post distinguishes two scores. Option probabilities rate each available choice and add up to one. Confidence is a separate score for the decision as a whole. In one example, purchaseProduct had a probability of 0.84 while the decision's confidence was 0.81. Every threshold in the post uses confidence.

The test: three agent versions, 900 turns

The test agent is Wonder Toys, a demo toy store with five tools: search the catalogue, show one product, buy it, check an order, or cancel one. Arize wrote 100 messages of the kind a shopper would type and ran each through three versions of the agent, three times each, for 900 turns. All versions share the same prompt, tools and chat model. Only the tool picker differs.

• As shipped: the chat model picks its own tool.
• Always enforced: the decision model, here gpt-6-luna, picks and the chat model must call that tool.
• Gated at 0.7: the decision model's pick is used only at a confidence of 0.7 or higher. Below that, the chat model chooses itself.

On the headline numbers, the first tool matched Arize's label in 78% of turns for the shipped agent, 95% when enforced and 91% when gated. The judge rated tool calls right 79%, 89% and 92% of the time. Median time per turn was 6.5 s, 5.6 s and 7.1 s, and cost per turn was $0.0217, $0.0210 and $0.0220. Arize notes that part of the enforced version's lead comes from how it scored purchases, which the post examines later.

Low-confidence picks were the weak spot. Across the 100 test messages, a pick with confidence below about 0.7 led to a wrong turn one time in three. Between 0.7 and 0.8 it was about one in six, and above 0.8 fewer than one in ten. Arize says the threshold is specific to its setup. Under the gate, empty or unfinished replies fell from 21 to 3 across 300 turns per version, and the judge rated tool calls as correct as often as before. The original agent had none in this test.

Two example turns

For the message "Cancel my most recent order", the router leaned toward using no tool at all (0.58), with the order lookup at 0.26, and its confidence was 0.50. In the always-enforced version the chat model could not call a tool, so it wrote "Let me look up your orders first" and stopped, three times out of three. In the gated version, 0.50 fell below 0.7, so the chat model chose for itself, called checkOrderStatus and found the order.

A purchase message for toy-076, with a delivery address in Pune, shows the other case. The router chose purchaseProduct with a confidence of 0.81, so the gate stayed shut. The chat model filled in the product ID and address, placed the order, and the judge graded the turn right every time.

Grading picks with Phoenix annotations

A confidence score shows how sure the router was, not whether the turn went well, so Arize graded every turn and wrote the grades onto the trace as annotations. Phoenix supports human, LLM judge and code-check annotations. Arize used the last two.

• LLM judge: Phoenix's tool-selection evaluator reads the request, the available tools and the agent's calls, then says whether they were right. Each turn was graded three times, and the verdict at least two runs agreed on was used.
• Code check: before the run, Arize recorded the tool it expected first for each message. The check passes if the first tool actually called matches.

Arize says this makes a failed turn self-diagnosing. If both grades say no, the pick was the problem. If only the judge says no, the pick was fine and the fault lies in the arguments or the reply. Because the grades are annotations, a project can be filtered on either one to find every turn with the same problem. For teams that want to repeat the comparison, Arize points to Phoenix datasets and experiments, where inputs and expected tools go into a dataset and each agent version runs as an experiment.

What it means for agent builders

For anyone running a tool-calling agent, the post offers a concrete pattern: trust the router's pick at 0.7 or higher, otherwise let the chat model choose as it always did. The 0.7 cutoff is Arize's own, drawn from its 100 test messages, and the company says teams should find the threshold for their own agent. The results also come from a demo store with five tools, so they show a method rather than a guaranteed outcome elsewhere.