LangChain Grows Self-Serve Analytics Questions 40x With Layered Agent Context

Context as the foundation for an analytics agent

Emily Hawkins, head of analytics engineering at LangChain, and Logan Cochran, analytics engineer at LangChain, explained how their team keeps an AI analytics agent accurate while the business keeps changing. Their starting premise, as summarized by dbt Labs, is that an agent is only as smart as the context it is given. A language model has no built-in understanding of a company's data, business rules or internal vocabulary.

The first layer is the dbt project itself. LangChain syncs table and column descriptions from dbt into BigQuery, so Hex and any other internal agent reading from BigQuery inherits the same descriptions. The Hex agent also has direct access to the dbt repo and DAG. That lets it reason about upstream and downstream dependencies and the SQL logic behind a model, not only column names.

The team had to change how it writes documentation. A vague description such as "the status of the account" was acceptable when only expert humans read it. For an agent, the post says the description needs to be closer to a definition of the current lifecycle status of the account in Salesforce, where active means an active paid contract, churned means a paid contract that has ended, and prospect means the account is not yet a customer.

Three more layers inside Hex

On top of the dbt layer, LangChain adds three more pieces inside Hex:

• Workspace guides: markdown files that explain business concepts an agent needs to interpret a question correctly, such as how the team measures ARR or its internal product vocabulary. The team writes them in its dbt repo and imports them into Hex through a GitHub Action, so they stay version-controlled and available to other agents.
• A semantic layer: metrics and join relationships defined in code for the team's most important models, so a question about a revenue metric resolves the same way each time it is asked.
• Endorsements: markers for which mart models and dashboards are trusted sources of truth. LangChain is deliberately selective, since an endorsement that is not a strong signal stops meaning anything. Any change to an endorsed dashboard requires a data team review before it publishes.

A changed role for the data team

According to the post, this combination changed how the team sees its own job. Rather than trying to answer every question that comes in, the data team now focuses on owning the context that lets other departments self-serve directly. The team illustrated the shift with a before-and-after view of departments routing questions through the data team versus querying the agent directly.

The speakers were candid that the system works even though the business changes faster than documentation can keep up. A pricing model change can cascade across multiple domains, and a business domain does not need airtight metric definitions before an agent can start being useful. The team treats context as something to document continuously, not a project with an end date.

What the 40x figure rests on and what comes next

LangChain says the system increased the number of questions asked through self-serve analytics by 40x. It attributes that to three things: continuing to document context as it changes, educating users on how to work with the agent, and measuring impact to iterate on what context is missing.

The roadmap includes adding evals and using Hex's CLI to enrich context further. For other data teams, the account points to documentation written for machine readers, version-controlled business guides and selective endorsement of trusted assets as the practices behind LangChain's results. The 40x figure is LangChain's own claim, and the post does not give underlying question counts.