FAR.AI Introduces SynthDeception Dataset for Testing Deception Probes
FAR.AI has introduced SynthDeception, a synthetic dataset designed to test probes that detect falsehoods, deception and sabotage in realistic agentic scenarios. The company says the probes perform well on SHADE-Arena and BashArena, although they do not match probes trained on its larger FIBS dataset.
- SynthDeception is a synthetic dataset for studying falsehood detection probes.
- Its scenarios cover 17 industry domains and use 595 tools.
- Scenarios range from roughly 1k to 100k tokens.
- SynthDeception probes perform well on SHADE-Arena and BashArena.
- FAR.AI says dataset scale and label verifiability affect probe generalization.
A synthetic alternative to the FIBS dataset
In its November 9, 2026 blog post, FAR.AI described SynthDeception as an attempt to measure how far falsehood detection probes can get when trained on data produced by one synthetic generation pipeline. The approach differs from FIBS, or Falsehoods In Broad Settings, which FAR.AI previously used to train probes with a dataset of 163,016 rows assembled primarily from other publicly available datasets.
SynthDeception instead contains realistic scenarios covering deceptive and non-deceptive agent behavior. Each scenario is paired with follow-up questions and honest or dishonest answers. These questions are intended to probe information contained in the scenario, including common knowledge, belief and intent.
FAR.AI says SynthDeception probes perform well on SHADE-Arena and BashArena, but do not close the gap with FIBS probes. The company attributes this in part to SynthDeception's smaller size, because generating a high-quality example is relatively costly.
Scenarios span industries, tools and context lengths
The dataset was built with NeMo Data Designer, a synthetic data generation framework in which datasets are organized as tables and attributes are generated through a directed acyclic graph of dependencies. FAR.AI varied the scenarios across 17 industry domains, including finance, energy, CBRN and education.
Half of the scenarios include tools and half do not. The dataset uses 595 tools in total: 163 real tools and 432 simulated tools. FAR.AI integrated 193 MCP tools that expose publicly available, permissively licensed APIs, then added a synthetic tool simulator for situations requiring more context-specific capabilities.
Scenarios range from roughly 1k to 100k tokens. The pipeline also sampled user personas from Nemotron Personas USA, described by FAR.AI as an open corpus of millions of synthetic personas with detailed demographic and personality attributes.
Why label verifiability matters
FAR.AI defines deception as intentionally trying to cause another party to hold a belief that the agent does not believe to be true. Its taxonomy separates deception into hiding reality from another party, called dissimulation, and showing a false reality, called simulation. The dataset also includes confusing forms of non-deception that could be mistaken for deceptive behavior.
The company reports that probes trained on subjective questions with answers that are difficult to elicit or infer generalize noticeably worse. This includes questions about intent. FAR.AI says the finding supports FIBS's use of contextual falsehoods as a training target, because contextual labels can be established from evidence in a scenario.
When FIBS probes were evaluated on SynthDeception, however, they performed better at larger model scales on every question subset, including intent. The results point to two factors for probe development: the scale of the training set and whether its labels can be verified from available evidence.
