dbt Labs: Semantic Layer Lifts AI Analyst Accuracy to 96-100% in Benchmark

The benchmark and the argument behind it

Noonan used his dbt Summit 2026 session to trace the long history of failed self-service analytics. He started with a 1971 natural-language moon-rock query system, moved through 1990s OLAP cubes, and covered a string of more recent natural-language BI features that never quite stuck.

His case for why the current attempt differs is that large language models have solved the "teaching the machine your words" half of the problem. What remains is grounding, which means teaching the machine what those words actually mean inside a specific business.

The benchmark came from dbt Labs' own developer experience team. The same model scored 96 to 100% accuracy when answering through a semantic layer, and 80 to 91% when writing raw SQL directly against a database schema. Noonan framed the gap bluntly: every point of difference is a confidently wrong number landing in someone's Slack channel.

Three ways to ground an AI analyst

Noonan described three converging approaches teams use to give an AI analyst context:

• A semantic layer with pre-defined metrics.
• An ontology-style model of entities and relationships, in the style of Palantir.
• An agent context layer that assembles relevant context at query time. He described this as the direction Fivetran and dbt are building toward together.

He also drew a split within context engineering. "Just in time" context is stable metadata such as a semantic layer or agent skills, which does not change much. "Just in case" context is the search infrastructure needed for high-volume unstructured sources like call transcripts and support tickets.

Fundamentals and trust

Most of the talk, according to dbt Labs, focused on what Noonan called the unglamorous fundamentals that separate a demo from a durable system. These include data quality and observability practices such as tests, freshness checks and anomaly detection, documentation written to be read by machines as much as by people, and consistent data modeling.

On trust, Noonan recommended keeping the chat interface in a public channel like Slack, so mistakes and corrections happen visibly rather than in a private thread. He also cautioned against relying too heavily on "LLM as judge" evaluation, because a judge model shares the same blind spots as the model it is grading.

He returned to two lines: "the worst thing that you can do is give somebody a confidently wrong answer," and "silent failures are the killer in any IT system."

Adoption and what it means for data teams

Noonan cited a figure that roughly half of generative AI pilots are abandoned after proof of concept. He also pointed to this year's State of Analytics Engineering report, in which about half of dbt users said they are actively building toward conversational analytics.

For data teams, the message of the session is that accuracy depends less on the model alone than on the context around it. Defined metrics, tested and documented data, and visible error correction are presented as the basis for moving from pilot to production. The benchmark is dbt Labs' own, and the source does not detail its methodology.

Noonan closed with a live demo of a personal AI trading assistant built on dbt's MCP server, which pulls SEC filings and technical indicators. He noted that a trade it had recommended had closed profitable the day before his talk. Sessions from dbt Summit 2026 are available on demand.