Microsoft Builds Synthetic Companies to Test Copilot Without Customer Data

The problem: realistic data without real data

Microsoft says evaluating Copilot at enterprise scale requires test data that captures the full complexity of real organizations, including reporting structures, project histories, communication patterns and domain specific workflows. Real enterprise data, however, contains sensitive information that cannot be used for evaluation under any circumstances.

To resolve this, the company built a platform that rests on three pillars: synthetic enterprise environments, automated query generation pipelines, and structured quality frameworks. Microsoft frames the goal as answering two questions: whether Copilot is doing what users need, and whether that can be proven without ever touching their data.

How the synthetic environments work

Instead of small, manually curated test fixtures, Microsoft generates full-scale simulated workspaces. Each one is called a tenant, which the company defines as an isolated organizational environment with its own users, data and configurations. The environments include realistic users, meetings, emails, documents and organizational structures.

To keep them realistic, Microsoft grounded them in publicly available sources:

• O*NET data from the U.S. Department of Labor, used to make employee roles and responsibilities realistic, calibrate the mix and frequency of work activities, and tailor workflows such as clinical trial reviews or portfolio updates.
• Online job postings, which provide role titles, reporting hierarchies and departmental compositions.
• SEC filings, which inform organizational scale, business unit structures and financial contexts.

Microsoft says all of these sources have been reviewed and approved by its legal and policy organization. The environments are also not static. They simulate organizations that change over time, with meeting histories, project timelines, email threads that develop over weeks and organizational changes. According to Microsoft, this matters because many Copilot queries are time aware, such as asking what the team decided last Thursday or what has changed since the last quarterly review.

Three environments have been generated so far: a pharmaceutical company with regulatory submission workflows and safety review meetings, a capital markets firm with deal pipelines and compliance communications, and an IT/software company covering sprint planning, code review and product launch coordination.

Automated queries and quality checks

Synthetic environments still need realistic questions to test Copilot. Writing thousands of grounded, industry specific queries by hand is, in Microsoft's words, prohibitively expensive and slow. The company therefore built a six-stage automated pipeline. It starts with a large library of common workplace tasks, narrows them to what is relevant for an industry, and adapts them with domain specific language. AI then generates queries grounded in the environment's data, rewrites them to sound more natural, and validates each one for accuracy and uniqueness.

Quality is assessed with an internal semantic evaluation framework. Microsoft lists structural fidelity, contextual relevance, behavioral realism and content safety as the dimensions. The content safety check asks whether generated material is free of harmful, offensive or personally identifiable content and avoids impersonating real individuals.

Microsoft also points to automation as a way to address practical limits: manual processes give subtly different results each time, the number of industry verticals and configuration variants exceeds what a team can build by hand, and manual creation takes days, slowing feedback.

What it means

Microsoft says the platform helps establish the first standardized evaluation baseline for Copilot. For organizations that use Copilot, the stated benefit is evaluation that reflects industry specific work, such as pharmaceutical or capital markets scenarios, while keeping customer data out of testing. The company describes the environments as scalable baselines that teams can build on as they see fit.