Data Culture lifts AI analytics agent accuracy from 0% to 94.7% by adding context
Data Culture, a boutique data and AI consultancy, says it raised an e-commerce analytics agent from 0 percent to 94.7 percent accuracy by adding layers of context, while the underlying model stayed the same. Director of analytics engineering Opeyemi Fabiyi presented the three-step evaluation framework at a dbt Summit 2026 breakout session.
- Data Culture presented a three-step evaluation-driven development framework for AI analytics agents at dbt Summit 2026.
- On an e-commerce deployment, accuracy rose from 0 percent to 94.7 percent using the same model and added context layers.
- The healthcare deployment also passed Data Culture's internal 90 percent production threshold.
- Evaluation combines a four-tier result comparison with a deterministic layer (about 80 percent of questions) and an LLM judge (about 20 percent).
- The evaluation pipeline and golden question templates are open sourced on GitHub.
Why the team built an evaluation framework
Data Culture spent its first five years on classic modern data stack work, including data modeling, BI migrations and platform migration. Over the past year, according to the company, client demand shifted toward operationalizing workflows with AI and letting stakeholders chat with their data. That pushed the team to develop its own methodology, evaluation framework and playbook for agentic analytics. It has used them on two production clients: a healthcare provider building clinical operations analytics and an e-commerce retailer building product and sales analytics.
The trigger was a question the team could not answer during its first client deployment: what is the accuracy of this agent? It had connected an agent to a Snowflake MCP server for a group of test users, but had no number to offer, not even a rough one.
The problem is not unique to Data Culture. dbt Labs' 2026 State of Analytics Engineering report found that trust remained the top concern for teams deploying AI agents to stakeholders. According to the report, 71 percent of respondents worried about hallucinated or incorrect data reaching decision-makers, and 83 percent named trust a top organizational priority.
Fabiyi borrowed the term evaluation-driven development from Chip Huyen's book AI Engineering and condensed it into one rule: don't ship what you can't measure.
Three steps: define correct, compare results, layer the checks
An early lightweight evaluation pipeline showed that several apparent failures were cosmetic. In one case the agent returned the right monthly gross revenue figures under a differently named column. In another it returned the correct three product categories plus a fourth row of legitimate uncategorized products. In a third it added one supporting column. A naive exact-match comparison marked all three wrong, although the answers were correct.
The resulting framework has three steps:
- Define what correct means. Teams write "golden questions" specific enough that two analysts would write the same SQL, then validate the expected answer with a domain expert.
- Define how to compare results. The comparison looks at the results of a query, not the query itself, because different SQL statements can give the same answer. Data Culture built a four-tier cascade: exact match, value-only comparison that ignores column names, overlapping-column comparison that tolerates extra context columns, and approximate-numeric match that tolerates rounding.
- Build layered evaluation logic. A deterministic layer handles roughly 80 percent of golden questions with fast, free, reproducible comparisons. A large language model acts as a judge for the remaining 20 percent, scoring groundedness, relevance to the question and comprehensiveness.
What moved accuracy
With the framework in place, the team ran controlled experiments using the same golden questions and the same model at every step. On the e-commerce deployment, accuracy rose as follows:
- 0 percent with no semantic context
- 50 percent after pointing the agent at modeled dbt tables instead of raw source data
- 52.6 percent after adding column descriptions
- 78.9 percent after adding business rules and default filters
- 94.7 percent after adding verified queries and named metrics
The healthcare deployment followed a similar curve. It started at 0 percent on raw data and eventually cleared Data Culture's internal 90 percent production threshold once business rules and verified queries were added. In both cases the gains came entirely from added context, not from a model change.
Fabiyi also pointed to Anthropic's published account of taking an agent from 21 percent to 95 percent accuracy after building a curated data foundation and an offline validation step.
What it means for data teams
Fabiyi argued that a dbt project's modeled tables and semantic context are the durable asset in any agentic analytics deployment, while the agent or LLM layer underneath can be swapped out later. For teams weighing whether an agent is ready for stakeholders, the results suggest that a measurable benchmark, not the choice of model alone, is what makes the answer to the accuracy question possible.
Data Culture has open sourced its evaluation pipeline, golden question templates, comparison logic and a technical walkthrough at github.com/datacult/dbt-agent-trust. Sessions from dbt Summit 2026 are available on demand.
