Databricks Says Genie Beats Frontier Agents on Enterprise Data Tasks
Databricks says its Genie data agent achieved a 91% pass rate in a new enterprise benchmark, compared with 45% and 58% for two frontier agents. Genie also used less than half the tokens and completed tasks three to four times faster, according to the company.
- Genie achieved a 91.0% pass rate in Databricks' benchmark.
- Agent A and Agent B recorded pass rates of 45.0% and 58.0%.
- Genie averaged 431,829 tokens per run, less than half the usage of either agent.
- The benchmark included enterprise assets beyond tables, including notebooks and dashboards.
- Databricks attributes Genie's results to business context surfaced by Genie Ontology.
Benchmark tests agents in a live workspace
Databricks developed the benchmark with Turing to evaluate business questions in a live Databricks workspace. The test measured three factors: answer accuracy, token usage and the time required to complete each task.
The benchmark used a fictional UK electricity and gas supplier called Calderwick Energy. Its workspace included hundreds of tables, with the largest containing around 120 million rows of half-hourly meter data. It also included notebooks, scheduled jobs, dashboards, saved SQL queries, code, metrics, documents and Genie Agents.
Databricks said the environment was designed to reflect the disorder of enterprise data work, including copied notebooks, abandoned analyses, old documents, development copies, frozen reporting tables and dashboards that had not been touched in months. Agents ran each task 5 times against the same workspace.
Genie led on accuracy, speed and token use
Genie recorded a 91.0% pass rate, compared with 45.0% for frontier Agent A and 58.0% for frontier Agent B. Its average wall time per run was 100.3, compared with 434.6 for Agent A and 306.5 for Agent B.
Genie also averaged 431,829 tokens per run. Agent A used 889,019 and Agent B used 1,051,070. Databricks said Genie was 4x faster than Agent A and 3x faster than Agent B, while using fewer than half the tokens on average.
For the evaluation, Genie ran natively. The other agents connected through a Databricks toolbelt, which excluded the Genie MCP because it would have given them access to the Genie capabilities being tested. All agents received the same questions, and answers were assessed using a calibrated grading rubric with predefined limits for acceptable numerical differences.
Why enterprise context changed the results
Databricks attributes Genie's performance to business context extracted by Genie Ontology. The company describes Genie Ontology as an enterprise context layer that automatically extracts and organizes knowledge from assets such as notebooks, dashboards and saved SQL queries.
Databricks argues that table schemas alone often cannot establish the definitions and calculation rules needed to answer business questions. Those rules may be recorded in a notebook, embedded in the SQL behind a dashboard or preserved in a saved query.
One benchmark task asked agents to identify Calderwick Energy's historical Commercial planning rate for FY24/25. The table showed 0.31 pence per kWh, but a source notebook documented the historical rate as 0.52 p/kWh and explained that the table used FY25/26 assumptions for earlier billing periods. Databricks says Genie Ontology surfaced the relevant historical information and linked it to the source notebook, allowing Genie to return 0.52 p/kWh in its first response across all 5 runs.
Databricks says teams can replicate the benchmark in their own workspace by contacting their account team. The company also directs readers to its GA announcement blog and official documentation for more information about Genie Ontology.
