Checkr Cuts Metrics Investigation From Hours to Minutes With AI Context

The problem: a different wrong answer every time

Fischbach works with a small central analytics engineering team and roughly 50 analysts embedded across the business. Checkr's Snowflake instance and dbt project span more than 6,000 models. When she connected an AI agent directly to Snowflake through an MCP and asked the same question several times, she got a different wrong answer each time. She called the pattern metric drift.

She illustrated it with FRT (First Reply Time), the core SLA Checkr's operations team is measured against. Asked "what queues drove FRT below 90%?", the agent did not know what FRT meant, what a queue was, or what the 90% threshold referred to. It could only see field names and table structures, not the business logic behind them.

Three layers of AI-readiness

Fischbach's framework splits AI-readiness into three layers:

• The data warehouse: the dictionary of every data point that exists.
• The semantic layer: the grammar that defines valid joins, calculations and metrics, and prevents metric drift.
• AI context: the conversational bridge that, in her view, most teams skip entirely.

Omni's version of the third layer consists of five structured fields: synonyms that map business language to field names, sample values so an agent can parse natural language into valid filters, field descriptions (often already written in a dbt project), free-text behavioral guidance for edge cases, and pre-built sample queries for complex, multi-step questions.

Her core principle is to reuse context that already exists in Confluence pages, meeting transcripts, PRDs and Slack threads rather than write new documentation from scratch.

An automated feedback loop

To fill in that context at scale, Checkr built an automated loop. A golden set of evals, made of real stakeholder questions paired with the correct query plan, runs against Omni. A Claude-based agent clusters the failures. A second skill pulls relevant context from a vector store of company knowledge and drafts a fix. A third skill re-runs the failing evals to check that the fix worked, then opens a merge request for a subject-matter expert to review.

A simpler manual version exists too. A team can paste a Confluence page or meeting transcript into an AI assistant together with the relevant dbt model code, ask it to extract synonyms and behavioral guidance, then review the output and add it to Omni by hand.

The result and a caution on over-adding context

The payoff came when Christian, one of Checkr's operations leads, asked the agent what queues drove FRT below 90%. With synonyms, channel-specific SLA targets and edge-case rules already encoded, the agent identified the underperforming week and broke it down by queue and channel. It ruled out the high-volume queue everyone had assumed was the culprit and pointed to international queues instead. In Christian's words: "What would have taken hours got done in minutes, and with enough directional clarity to actually change how we responded."

A data professional at OutSchool raised a warning from the audience. Adding AI context broadly, without a specific failure to fix, made their agent's answers slower and roughly doubled token costs without improving accuracy. Fischbach and Merrick's advice matched Checkr's experience: match the fix to the failure, prune fields nobody queries, and add context surgically where there is a measurable precision or recall problem, not as a blanket pass over every model.

What it means for data teams

The session suggests that connecting an agent to a warehouse is not enough on its own, even when a semantic layer is in place. According to the account, the missing piece was business language, thresholds and edge-case rules, much of which already sat in internal documents. It also points to a practical method: tie each piece of context to a failing eval, and have an expert review changes before they ship.