IBM's ALTK-Evolve lifts GPT-4.1 to the top of τ²-bench retail

Learned guidelines instead of weight updates

ALTK-Evolve builds a playbook of short behavioral rules from an agent's own attempts. According to the IBM Research authors, it analyzes the agent's attempts on training tasks and uses the model to distill successes and failures into rules such as how to authenticate a customer, confirm a transaction, or handle a return. These guidelines are then appended to the agent's system prompt for later conversations.

The same model, GPT-4.1, powers both the guideline extraction and the agent. The authors state that no weight updates or human annotation are needed. They previously demonstrated the approach on coding agents and now test it on customer service, where success requires following domain policies, understanding a request and executing the right tool actions.

How the test works

τ²-bench, from Sierra, pairs an agent with an LLM-simulated customer who has a goal, a set of domain tools and a written policy document. A task counts as solved only when the final database state matches the expected outcome, a strict binary reward. IBM used the retail and airline domains, the benchmark's own basic ReAct agent with a THINK then ACT loop, gpt-4.1 as the agent model, gpt-5.5 as the customer simulator and 4 trials per task. The only change was the block of learned guidelines added after the domain policy.

The team compared three setups: a baseline with no guidelines, sup3, which keeps only guidelines extracted from at least three distinct training tasks, and lossless, which keeps every guideline retained after consolidation. Retail had 31 guidelines in sup3 and 292 in lossless. Airline had 14 and 101.

Two metrics were tracked: Mean Accuracy, the average success rate across the four trials, and Pass^4, the share of tasks passed in all four trials. Pass^4 is the metric the leaderboard reports and measures reliability.

Results on held-out tasks

• Retail (40 test tasks): baseline 78.1% Mean Accuracy and 57.5% Pass^4. Sup3 reached 78.8% and 60.0%. Lossless reached 82.5% and 67.5%.
• Airline (20 test tasks): baseline 57.5% and 35.0%. Sup3 reached 60.0% and 45.0%. Lossless reached 65.0% and 40.0%.

The best amount of guidance differed by domain. In retail, more guidelines were better on every metric. In airline, lossless led on accuracy (65% against 60%), but the compact sup3 set was more reliable, with Pass^4 of 45% against 40%. The authors conclude that support count is a knob to tune per domain rather than a value to hard-code.

Cost and conversation length

The guidelines add cost, which lands in prompt input tokens. Total LLM cost per conversation, covering the agent and the user simulator, rose in retail from $0.084 at baseline to $0.095 with sup3 (+12%) and $0.139 with lossless (+65%). In airline it rose from $0.081 to $0.097 (+19%) and $0.107 (+32%).

The agent does not get there by talking longer. On matched runs solved under both settings, assistant turns per conversation were slightly lower with guidelines. In retail, baseline and lossless runs averaged 10.59 and 10.16 turns. In airline, they averaged 9.15 and 8.82.

What it means

According to IBM Research, the results show that a basic agent on an older model can gain both capability and reliability from learned memory alone, without a frontier model or any model retraining. The trade-off is higher prompt cost, and the best guideline set size depends on the domain. The airline test set had only 20 tasks and retail 40, so the results come from small held-out samples.