Spotify study: simple pairwise models beat Transformers on 4 of 5 benchmarks

What the study asks

Sequential recommendation models predict what a user will interact with next. Transformer architectures such as SASRec are widely used for the task, because self-attention lets them model complex, context-dependent relationships across a user's interaction history. The field commonly evaluates them on benchmarks built from datasets such as Amazon and MovieLens.

Spotify Research asks what these benchmarks actually test. According to the researchers, strong predictive signals can already come from relatively simple patterns: item popularity, pairwise item-to-item transitions and recency. If simpler models built on those signals match more expressive ones, the benchmark may not be measuring what is assumed.

Three levels of model complexity

The team compares three levels of what a model can represent, and evaluates seven models in total.

• Last-item models: only the most recent interaction contributes to the score. The three controls are a Markov Chain (MC) model, FMC and FMC+. FMC learns to separate the observed next item from one sampled alternative, while FMC+ compares it with the whole catalogue during training.
• Pairwise models: every item in the history contributes evidence about what follows it, but each contribution is kept separate and combined with fixed recency weights. The two models are Sequential Rules and the newly proposed Probabilistic Collaborative Transition Model (PCTM).
• Transformer-based models: SASRec and eSASRec, an enhanced version presented at RecSys 2025, can interpret each history item in the context of the others. The researchers call this capacity higher-order sequence modelling.

PCTM counts which items follow each source item in training histories, weighting nearby occurrences more heavily. It starts from equal probabilities for all catalogue items and updates them with observed counts, so scarce data leans on the starting assumption. According to the researchers, its estimates come directly from event counts, without item embeddings or a sequence encoder.

Evaluation setup

The models were tested on Amazon Beauty, Sports and Toys, and on MovieLens-1M and MovieLens-20M. All used the data splits and evaluation protocol released with eSASRec. The researchers say their SASRec and eSASRec reproductions closely matched the scores in the eSASRec study, differing by at most 0.0033 NDCG@10 across all five datasets. NDCG@10 rewards a model when the item the user interacted with next appears nearer the top of its first ten recommendations.

Results

On four of the five benchmarks, at least one pairwise model scores higher than both Transformer reproductions. On MovieLens-20M, however, the tested Transformers keep an advantage over the pairwise models.

Earlier history clearly helps. On every dataset, both history-aware pairwise models beat all three last-item controls, and the stronger pairwise model improves on the best last-item control by approximately 20 to 46%, per the researchers. FMC+ is already competitive: it scores close to SASRec on Beauty and above both Transformers on Toys.

Both pairwise models score above both Transformers on all three Amazon benchmarks. On MovieLens-1M, PCTM also beats both, reaching 0.1815 against eSASRec's 0.1739, a relative improvement of about 4.4%. The researchers stress that these are point estimates and that the paper does not report confidence intervals.

What it means

For people building or evaluating recommenders, the finding is a caution about reading benchmark results. A high score on these Amazon and MovieLens tests does not by itself show that a model captures joint interactions within a user's history, since models that treat each past item separately can match or exceed Transformers. MovieLens-20M is the exception in this study, where the Transformers keep their lead. The source text provided ends before the full discussion of the remaining results.