Artificial intelligence news.
Straight from the original announcements.

KS&R Report Finds Synthetic Data Succeeds in Simple Test, Struggles in Complex One

November 12, 2026, 13:00By the AI News Bites Editorial TeamSource: PR Newswire
KS&R Report Finds Synthetic Data Succeeds in Simple Test, Struggles in Complex One
Image: PR Newswire

KS&R has released an independent evaluation of four synthetic data providers, finding that all matched a real data baseline in a simple study but fell outside it in an initial complex study. The report uses the consultancy's ECHO Index to assess how faithfully synthetic data reproduces human responses.

Key points
  • KS&R evaluated production models from four synthetic data providers.
  • All four matched the baseline in the simple study.
  • All four initial submissions fell outside the baseline in the complex study.
  • The report is based on two datasets, not a ranking of providers.
  • KS&R plans to extend the ECHO Index to digital twins and synthetic personas in Q1 2027.

Four providers tested against real data

The State of Synthetic evaluates production models from LagrangeDATA, Livepanel, Simulacra and Lewsearch. Each provider generated a complete synthetic copy of two public datasets, one simple and one complex.

KS&R scored every submission against a fit-for-purpose baseline generated from the real data alone. The evaluation focused on sample boosting, where a model generates new respondents for questions that real people have already answered.

According to KS&R, all four providers landed within the baseline for the simple study. In the complex study, however, all four initial submissions fell far outside it.

The report examines where models failed

KS&R says the full report shows where each model broke down and how targeted changes moved submissions closer to the baseline. It also describes cases where a provider's subject-matter expertise made a measurable difference on particular questions.

The findings apply to specific submissions on two datasets. KS&R says they are not a ranking of the firms and do not amount to a verdict on synthetic data as a whole.

The consultancy created the ECHO Index, described as a patent-pending evaluation methodology, to give buyers a neutral way to assess synthetic data. The firm says it has no synthetic products to defend, while providers often validate their own models and present a single score as evidence.

“Synthetic data is viable when it is governed. Don't buy the promise. Buy the evidence,” Ben Cortese, PhD, Senior Vice President, Decision Sciences & Innovation at KS&R, said in the announcement.

A framework for governed synthetic studies

The report arrives as insights teams face pressure to study hard-to-reach audiences, work faster and operate with tight budgets. KS&R says synthetic data is being sold faster than governance can be developed, making independent evaluation a central concern for buyers.

The report gives research leaders guidance on running a governed synthetic study, including how to budget for iteration and which questions to ask providers before purchasing. KS&R recommends that buyers have synthetic data evaluated by a neutral third party before relying on it.

The executive summary and full report are available free from KS&R. Providers can have their models scored and listed in the ECHO Registry, while research and insights leaders can work with KS&R to assess where synthetic data fits, test data they have already purchased or check a model before committing to it.

KS&R is continuing the evaluation and plans to extend the ECHO Index to digital twins and synthetic personas, with completion planned for Q1 2027. For users of synthetic data, the results support a more conditional approach: success on a simple task does not establish performance on a complex one, so testing and governance remain part of the buying process.

KS&RLagrangeDATALivepanelSimulacraLewsearchresearch findingsbenchmarksprivacy

Related news

JetBrains asks 12 AI models for 1,200 programming hot takes
ResearchNovember 15, 2026, 18:02

JetBrains asks 12 AI models for 1,200 programming hot takes

Twelve AI models overwhelmingly attacked Python, Rust and TypeScript in a JetBrains experiment that collected 1,200 opinions about programming languages. The models also…

Hugging Face page lists XYBench without details of the benchmark
ResearchNovember 13, 2026, 18:02

Hugging Face page lists XYBench without details of the benchmark

Hugging Face has listed an AllenAI page titled “XYBench: Helping with the problem behind the question.” The supplied page text does not describe the benchmark, its…

Meshy.ai benchmark finds fewer print failures with Make Printable
Products and toolsNovember 13, 2026, 16:00

Meshy.ai benchmark finds fewer print failures with Make Printable

Meshy.ai says its Make Printable feature reduced the share of AI-generated 3D prints with major missing features or holes from 44% to 8% in an FDM benchmark. The study…