Meta Deploys LLM-Generated Tests That Catch Bugs Before Code Lands

Tests designed to fail on bad code

Meta's engineering team describes Catching JiTTests as a different approach from most AI-based bug prevention. Many tools use LLMs or agentic AI to review code and flag faults based on prior experience, working through the same process a human reviewer would, only faster. Catching JiTTests instead generates executable tests that demonstrate a concrete behavioral difference introduced by a change.

According to Meta, this requires more computational steps, but it catches classes of bugs that LLM- or agent-based code reviews might not. The tests are automatically generated and designed to fail, surfacing bugs in code changes (diffs) before they land in production. Meta says the scale of its codebase, with hundreds of millions of lines of code constantly evolving, means traditional testing can't keep pace.

Two workflows and a filter for false positives

The main challenge at Meta's scale is separating strong catches, meaning true-positive failures that reveal real bugs, from weak catches that may be false positives. Meta built two workflows to address this:

• An intent-aware workflow that infers the intention and risks of a code change and targets test generation toward likely failure points.
• A dodgy diff workflow that treats every code change as if it were a mutant or bug and generates tests to expose its behavioral differences without reasoning about intent.

The system runs overnight on spare capacity and targets diffs from the previous day that are deemed risky, using a mechanism similar to Diff Risk Score (DRS) trained on previous code changes. Tests that fail on the actual diff are harvested as weak catches.

To filter them, Meta combines an ensemble called LLM-as-Judge, which aggregates judgements from several state-of-the-art LLMs, with a rule-based assessor called RubFake (Rule-Based False-Positive Killing Environment). Meta says LLM-as-Judge has rich semantics but its judgements are non-deterministic and challenging to verify, while RubFake provides verified, deterministic judgments. Together they produce a ranking of weak-catch candidates.

Human-on-the-loop instead of human-in-the-loop

Based on that ranking, the system decides whether to contact a human engineer. Rather than having an engineer review every output, Meta uses what it calls a human-on-the-loop approach: engineers set criteria and review outcomes at an aggregate level. The goal, according to Meta, is to keep the consistency of human labeling while reducing the likelihood that an engineer is pulled in to review a weak catch.

Meta says this means engineers rarely need to read test code and only need to confirm whether an observed behavioral change was intended.

Results and what they mean

As of the writing of the post, 589 engineers at Meta, or 64% of its outreach, have engaged with Catching JiTTests. Those engineers reported genuinely actionable findings, with a 60% rate of true positives. In the first half of 2026, Meta says Catching JiTTests played a key role in catching 74 serious (SEV) privacy bugs at the diff stage, and lowered the overall occurrence of privacy SEVs by 20% in that period.

Meta argues the result matters as agentic software development lets developers write, review and ship code faster, which raises the need for faster testing. The company says it has shown LLMs can act as an additional safety layer for what traditional tests miss, and that AI-generated testing can scale enough to reduce organizational risk. It calls the approach generalizable: engineers define success criteria early, let AI execute the testing, monitor outcomes in aggregate and intervene at the policy level.

The work is detailed in the paper "Just-in-Time Catching Test Generation at Meta," which was presented at FSE 2026 and featured at Product@Scale.