Wharton study: one review can shift an AI shopping agent's pick by 99 points
A single outside review can change an AI shopping agent's product choice by as much as 99 percentage points, according to new research from the Wharton School. The study finds that which sources an agent sees, the order it sees them in, and how its tools deliver them can all substantially influence what it buys.
- A single outside review changed one agent's product choice by as much as 99 percentage points, in the Wirecutter condition.
- Showing two or three conflicting sources did not produce an averaged choice; Wirecutter-containing combinations often still favored its recommended product.
- Reordering the same three sources significantly changed choices for four of six models, while Claude Haiku 4.5 showed no significant order sensitivity.
- A memory of "I love hiking!" pulled four models toward a much more expensive Garmin despite a cheaper, better-rated option.
- Bundled versus sequential tool calls produced significantly different choices in all four models tested.
What the researchers tested
The study was conducted by Anushka Kumar, Lennart Meincke, Dan Shapiro, Lilach Mollick, Stefano Puntoni and Ethan Mollick, with affiliations spanning the Wharton School, WHU Otto Beisheim School of Management and Glowforge. It was published through the Wharton AI & Analytics Initiative.
Using ACES, a mock e-commerce environment, the team ran five controlled studies in which frontier AI models selected a fitness watch. Each condition was run 200 times. Six models took part in the first four studies, while four took part in the tool-call study. The experiments separately varied single reviews, combinations and orderings of reviews, user-memory statements, and whether tool results arrived together or sequentially.
Reviews, order and memory change the outcome
Without prior source material, each model had a strong baseline preference among eight fitness watches. Showing the agent one review first, such as a Reddit thread, a Wirecutter roundup or a lifestyle listicle, changed selection rates by as much as 99 percentage points in the Wirecutter condition. The effects varied widely by model and source.
More sources did not produce a balanced choice. When agents saw two or three sources recommending different products, their choices did not average across the recommendations. Combinations containing Wirecutter often continued to favor the Wirecutter-recommended product, and for some models that effect became stronger.
Order also mattered. Reordering the same three sources significantly changed product-choice patterns for four of the six models. One other model showed a marginal effect, while Claude Haiku 4.5 showed no significant sensitivity to order.
Stored user information had an effect too. In a modified product grid, one watch was the cheapest, highest-rated and most reviewed. A previous user comment drawn from the system's memory ("I love hiking!") shifted four models toward a much more expensive, hiking-oriented Garmin. The other two models did not show the same response.
Even tool delivery matters
In the final study, four models received the same three recommendations either at once, as a "bundled tool call", or in three sequential calls. All four produced significantly different choice patterns depending on the method. Bundled delivery more strongly favored the Wirecutter-recommended product.
What it means for sellers and deployers
The researchers' takeaway is that AI shopping agents do not evaluate products in a vacuum. The same request can yield different results because of prior content, source order, stored user information and implementation choices that users may not see.
According to the study's authors, the results suggest that sellers may find agentic shopping harder to predict or optimize than traditional SEO. They also conclude that companies deploying shopping agents should test for consistency.
Wharton AI & Analytics Initiative: Your AI Shopping Agent Is Less Consistent Than You Think
