Netflix Builds Model to Measure the True Value of Recommendations

Why the usual metric falls short

A common first attempt at valuing recommendations is to ask what share of viewing comes from recommended titles. Netflix says this does not answer the causal question. Recommendations are not randomly assigned: a title appears on a member's homepage because the system predicts the member might want to watch it.

That means viewing a recommended title mixes two things, the effect of personalization and the member's underlying preferences. The authors describe this as the core measurement challenge, since recommendations are targeted to demand.

The obvious fix would be a large experiment that replaces the personalized experience with an unpersonalized one. Netflix argues this would be a poor member experience, because personalization is a central way members find valuable titles in a large catalog. Turning it off broadly is, in the authors' words, not an acceptable measurement strategy.

A day-by-day model of member choice

Instead, the team takes a hybrid approach. It builds a behavioral model that separates preferences from the incremental effect of recommendations, using natural variation in the recommendation system, such as algorithmic exploration. The work appears in the International Journal of Industrial Organization and is being presented at the ACM Conference on Economics & Computation (EC '26).

The model treats viewing as a daily choice. Each day, a member picks among the available titles or chooses to watch nothing. The target is the probability that a member watches each title on a given day, based on preferences, viewing history and the recommendations they see.

The authors say a simple model would be too restrictive, for example by assuming that removing one title shifts viewing proportionally to all others. In practice, the closest substitutes for a romantic comedy are likely other romantic comedies. To capture this, they use a low-rank choice model that learns representations of titles and of each member's current preferences from viewing behavior itself, rather than from manually specified features such as genre, cast or language.

Preferences are treated as dynamic. The model summarizes a member's past viewing into a current preference state, so someone who just finished a crime thriller can be in a different state than after a stand-up special or a kids' movie. According to Netflix, this design scales to millions of members.

The key component is an explicit recommendation effect, or "bonus," that raises the probability a recommended title is chosen. Netflix describes it as a reduced-form way to capture two mechanisms: recommendations create awareness, and they provide information by signaling that a title is likely a good match.

Validation against randomized experiments

The model assumes that, after conditioning on the member, title and state information available to the recommender, the remaining variation in whether a title is recommended is as good as random. The authors call this reasonable because they observe all the information the recommender uses, but they acknowledge it is still an assumption.

To test it, the team ran small-scale randomized experiments. Members were randomly assigned to treatment arms that slightly nudged recommendations toward and away from different categories of titles. This allowed the researchers to measure diversion ratios: when some titles become less visible, how much viewing moves to other titles and how much leaves Netflix viewing altogether.

The model was estimated on the control group, used to simulate the same changes, and compared with the experimental results. Netflix reports that the predicted diversion ratios had a correlation of 0.86 with the experimental estimates and an R-squared of 0.73.

What it means

With the model validated, Netflix says it can be used as an engine for counterfactuals: what happens if a title is not recommended, if a different set of titles is recommended, or if the current recommender is replaced with simpler alternatives. Because it separates member preferences, title appeal and recommendation effects, it can estimate both the impact on total viewing and the shift in which titles get watched.

The excerpt of the post provided ends as it begins to discuss counterfactuals on aggregate consumption and diversity, so those findings are not covered here. What the published description does show is a way to quantify the value of personalization without switching it off for members, by combining large-scale modeling with the credibility of small experiments.