Netflix adds AI to Pensive to diagnose failed data workloads

Why rules alone were not enough

According to the Netflix Technology Blog, practitioners at the company run millions of workloads each day across systems such as Titus, Spark, Trino, and Druid. On peak days, roughly 100,000 of them fail. Finding the useful signal in several logs, deciding whether the fault lies in user code or the platform, and choosing whether to retry, reconfigure, or escalate can eat engineering time and waste compute.

The first version of Pensive collected logs from failed workloads and used regular expressions and hand-authored Java rules to extract and classify errors. It then passed retry or remediation guidance to products such as Maestro. Netflix says this made about 50% of failures self-service and, in some cases, automatic.

The limit is coverage: a rule exists only after someone recognizes a new failure mode, reads the logs, writes the rule, and keeps it current as engines and log formats change. Over one three-month period, Pensive processed roughly seven million job-failure classifications, and about half fell into the unknown bucket.

Error extraction: the model picks line numbers

Pensive AI keeps the deterministic rules and wraps AI around them. Netflix says this gives broader coverage, low latency, controlled AI costs, and easier maintenance.

For error extraction, Pensive numbers each line of a log and gives the model a sequence of {lineNumber, text} records. The model returns only two integers, the first and last line of the error span, and Pensive rebuilds the text from its own copy of the log. Netflix explains that producing output tokens is much slower than reading input tokens, so this approach lowers cost and latency. Because the displayed text comes from the original log, it also reduces hallucinations, though the model can still pick the wrong range. For that reason extraction quality is part of the team's AI evaluations.

Extraction runs automatically whenever a Maestro step fails. It shares a request with the existing regex-based classification, and the whole response must return within Maestro's hard 90-second post-processing timeout. Logs are processed in parallel under a shared deadline set well inside that limit, and any per-log call that misses it is left out so the request still returns on time.

The data model also changed. The legacy format held one flattened error string, while AI extraction returns a list of errors, one per log source, with deep links to the original logs. Netflix kept both fields to avoid breaking clients. The Data Portal UI for failed Maestro steps reads the new field, while the query editor, Workbench notebooks, and some Slack clients still use the legacy path. The company says the migration is in progress.

On-demand debugging

Beyond extraction, Netflix built Pensive AI Debugging, which analyzes failed step attempts and offers root-cause analysis and suggested remediations. It is available through programmatic APIs, MCP, and the Data Portal UI, all behind a common entry point across the Netflix Data Platform.

A debugging response can take tens of seconds or a minute, compared with a few seconds for extraction. Netflix runs it on demand rather than automatically, which it says keeps costs under control because most failed steps are never opened. The failed step attempt, not the person who triggered the request, is the unit of work, so everyone looking at the same attempt sees the same diagnosis. Responses are cached by step attempt, so a second request becomes a key-value read instead of another inference call.

What it means for users

Netflix reports early results it calls encouraging. Failed step attempts where no one used Pensive AI averaged 321% more manual debug-log exploration actions than attempts where someone did. The company stresses this is an observational comparison, not a causal claim.

In practice, practitioners get a fast first look at every failure through extraction, and can go deeper with debugging when that is not enough. Netflix says it plans to explore more agentic debugging experiences next.