Netflix Cuts Synopsis Fact-Checking Cost by 52.4%

Why synopsis accuracy matters

A synopsis gives viewers a short preview of what to expect from a show or episode. Netflix uses both series-level synopses, which describe an overarching plot, and episode-level synopses, which describe individual episodes.

According to the Netflix Technology Blog, these descriptions must provide enough information to help viewers decide whether to watch while preserving some intrigue. They must also be factual. At Netflix’s scale, checking synopses by hand across thousands of titles, multiple episodes per season and multiple languages is not practical.

The company’s existing factuality service gives a reasoning model information about a title, including metadata and text assets, then asks it to assess a synopsis using a detailed rubric. Netflix says the previous method faced two major problems: expensive calls involving very long contexts, and explanations that often did not match the reasoning of human annotators.

Catching misleading descriptions, not only false facts

Netflix’s examples show that factuality involves more than identifying statements that are plainly untrue. A synopsis can contain individually correct details while still giving a misleading impression of a show.

For example, the company describes a synopsis focused on Lady Whistledown’s gossip sheet in Bridgerton. Although Lady Whistledown does publish such a sheet, the description could lead a new viewer to believe that the character is the central focus of the show, rather than Daphne Bridgerton’s experience in London’s marriage market.

Netflix also highlights descriptions that accurately mention Daphne and the Duke of Hastings but imply that their relationship begins as an immediate love affair. The company says this misses the fact that their relationship starts as a fake, mutually beneficial arrangement. Its system is intended to identify both these subtle errors and more obvious factual problems.

Confidence gates and specialized judges

The previous pipeline made five calls to a large reasoning model over contexts of more than 100k tokens for each judgment. Netflix’s analysis found that repeated calls were most useful for nuanced cases, where the model could reach different verdicts, while plainly true or false synopses usually produced consistent results.

The new pipeline first asks the model to provide a confidence level alongside its verdict and rationale. High-confidence cases can avoid unnecessary additional checks, while low-confidence cases receive more scrutiny. Netflix describes this as a confidence-gated version of its holistic model.

For harder cases, the company replaces identical repeated judges with specialized judges. The Literal Judge asks whether a synopsis is literally true. The Implicature Judge asks whether it creates a false picture for someone who has not watched the show. Netflix says this targeted approach works better than repeatedly querying one generic judge.

On Netflix’s custom benchmark, which contains synopses labeled by human annotators with factuality judgments and rationales, Recall-Fail rose from 0.77 to 0.86. The company also reports that explanations reached about 82% overlap with human rationales, up from about 69%. The results mean Netflix can check synopses more cheaply while making the system’s decisions and explanations more consistent with its expert reviewers.