Amazon Payments Uses Contextual Bandits to Personalize Its Acquisition Funnel

From generating content to choosing it

AWS frames the work as the next step after generative AI made personalized content cheap to produce. In an earlier post, the company showed how Amazon Bedrock can create personalized content at scale within brand guidelines and guardrails. The open question is now selection: among many options, which one should each customer see, and how quickly can a system learn the answer?

A multi-armed bandit (MAB) is a reinforcement learning method for settings with many options and limited traffic. Each content variation is treated as an "arm". The system tries arms on live traffic and gradually shifts impressions toward the ones that perform, while holding some back to keep testing the rest. AWS describes this as the trade-off between exploitation and exploration. Because it never stops doing both, a bandit keeps improving as new variations are added and does not need to wait for a test to end. AWS says A/B/n testing still has its place, but expects bandits to play a growing role as the number of variations rises.

LinUCB and the context vector

Amazon Payments chose the Upper Confidence Bound (UCB) strategy, which picks the arm with the highest estimated reward plus an uncertainty bonus. According to AWS, its deterministic selection rule gives an auditable, reproducible decision for every impression.

A standard bandit learns one best arm for the whole audience, so personalization needs context. AWS notes that segmented bandits run a separate instance per hand-defined group and each group needs its own traffic. Contextual bandits instead condition on a feature vector, so a pattern learned in one context carries over to similar visits. In production, each customer is represented by behavioral signals such as payment behavior and transaction mix. The entity ID is used only to route the recommendation back to the right visitor and is never a model input.

The team selected Linear UCB (LinUCB), introduced by Li et al. (2010). AWS calls it computationally efficient, auditable and able to handle a large arm space with limited warm-start data. It assumes the expected reward for an arm is a linear function of the context vector, which lets it generalize to unseen visitors. The exploration bonus is context-dependent: large for a kind of visitor an arm has rarely seen, small for one it has seen often.

Optimizing the whole funnel

The customer journey in this use case has three steps: application start, submission and approval. AWS says these stages do not move together. Content that boosts starts tends to attract a broad audience, but approval depends on whether the offer suits the applicant, so optimizing one stage can hurt another. AWS calls this the seesaw problem. Optimizing only for approvals has its own flaw, because approvals are rare and delayed, which starves the model of signal.

The solution runs one LinUCB model per stage and combines their UCB scores in a weighted linear combination. Weights can follow business priorities or be learned in a calibration step. The team used approximately equal weights. AWS suggests that others could weight approvals more heavily once the model is warmed, or use a Pareto frontier if the trade-off is contested.

On tuning, the exploration parameter alpha has a reasonable default of 1.0, according to AWS, with a typical range of 0.1 to 2.0. It advises starting higher when the arm space is large and history is limited, then lowering it as evidence builds. Values too high waste traffic on weak arms, while values too low risk locking onto a suboptimal one.

What the results mean

The reported outcome is mixed. One customer population gained a high single-digit percentage relative lift in final-funnel conversion, and another saw no improvement. AWS states that "The problem turned out to be the content, not the model." The implication from the source is that selection only helps when the available variations give the model something useful to choose between.

AWS also provides a code repository, including a quick start Jupyter notebook, a command-line demo and unit tests, so teams can try the method on synthetic data using Amazon SageMaker AI.