Microsoft Releases Playbook to Fix How AI Coding Agents Use Your Technology

Why agents get your technology wrong

According to Microsoft, asking a coding agent to build something with a product often produces code that looks reasonable and might even compile. On closer inspection, it can contain the wrong SDK version, a deprecated authentication pattern, or a setup nobody on the product team would recommend. Microsoft says this is not a random mistake: the agent did what its training data told it to do.

The company argues that developers increasingly let agents choose the SDK, the version and the pattern, and then judge a technology by the result. Waiting for better models is not a strategy, Microsoft writes, because a knowledge cutoff "tells you little about what a model knows about your product." What teams can change are the sources agents rely on: docs, MCP tools, skills, plugins, instructions, CLIs and APIs. The difficulty, according to the post, is knowing which change actually helps.

What the playbook covers

Since fall 2025, Microsoft's DevRel team has measured how agents perform with Azure, Cosmos DB, SharePoint Framework (SPFx) and Microsoft 365 Copilot extensions, using prompts similar to those real developers write. The findings were previously shared in the Agent Experience series of articles. The playbook combines them into one end-to-end method that others can run on their own technology.

• Results you can defend: Microsoft says evaluations can mislead. It reports seeing perfect scores for code that never compiled, and a "used Platform X" check that passed whether or not the agent used Platform X. The evaluation model describes what a result needs before it can be trusted, from criteria that judge meaning to gates that prove the code runs.
• Criteria that measure what matters: The playbook shows how to write criteria that a judge rules on consistently, how to calibrate them, and how to version them when a product changes. It also warns against traps such as letting a model write the criteria.
• From symptom to cause: A readout shows what failed, while the trajectory shows why. The playbook helps separate an extension that never loaded, one that loaded but was never called, and one that was called but applied wrong. Nine failure patterns that repeat across the technologies Microsoft evaluated each point to the surface to inspect first.
• Fixes that ship: The playbook explains how to test a proposed change as a hypothesis before shipping it, and how to bring evidence to the owning team, whether internal, a close partner or an open source project.

Microsoft says every step is grounded in scenarios run on Cosmos DB, SPFx and other technologies. Those evaluations led to dozens of shipped fixes to docs and agent extensions, including 46 improvements to the Azure Cosmos DB Agent Kit. The SPFx project upgrade is walked through end to end, and a quality checklist helps review an evaluation before it is run.

The AX Practitioner skill

Alongside the PDF, Microsoft released the AX Practitioner skill. Developers install it in their coding agent and ask questions while working on their own evaluation, for example how to word a criterion or why the agent ignored an extension. It can also review scenarios and criteria against the playbook's standards.

The skill answers only from the playbook. When a question goes beyond it, the skill says so and asks before using any other source. Microsoft reports that across 330 questions, the skill's answers scored 95% on average against what the playbook says.

Who it is for and what it means

The playbook targets anyone who builds technology or extensions that AI coding agents use: SDKs, APIs, services, CLIs, MCP servers, skills, plugins, instruction sets, or the docs behind them. People who advocate for a technology rather than own it can contribute scenario knowledge and the evidence that helps the owning team decide what to change.

The method does not depend on a specific evaluation system. Teams can use one they built or an existing one, and the playbook lists the capabilities to look for. For product teams, the message from Microsoft is that agent behavior is something they can measure and influence through the sources agents read. The playbook is available at aka.ms/ax-playbook. The post was written by Waldek Mastykarz, a Principal Developer Advocate at Microsoft.