Databricks Benchmark: Declarative Spark Pipelines Cut Agent Compute Time About 34x

Why agent-written pipelines raise the stakes

According to Databricks, over 70% of SDP pipelines on the platform are now created by agents. The company argues this puts pressure on two points: every attempt an agent makes costs compute and time, so iterations must be cheap, and code review becomes a critical gate, so the code must be succinct, readable and correct.

Apache Spark offers two programming models. A procedural pipeline tells Spark how to build each dataset, step by step, and the only way to know whether it works is to run it. A declarative pipeline describes the datasets wanted and leaves Spark to decide how to build them. Databricks compares this to prompting an LLM and letting it work out the method.

The company says a declarative pipeline can be checked for structural bugs before it runs, similar to compile-time checks in a typed language like Rust. Column references are resolved against the schema of the dataset they read from, dependencies between datasets are matched to a producer in the graph, and the graph is checked for completeness and ordering. Spark also handles execution order and saving results, so the agent writes less boilerplate.

How the benchmark was run

Databricks tested 22 data engineering tasks, each written like a real ticket that specified the table, column names and data types the pipeline had to produce. For each task it generated 12 input datasets containing messy values of the kind found in production data. The agent built a pipeline for every dataset with procedural PySpark and again with SDP, giving 264 builds per approach and 528 in total. All builds used Opus 4.8.

An automated grader that did not know which approach produced a result checked each output. Declarative pipelines first went through Spark's dry run, and the agent could revise a failed build up to a fixed limit. Databricks says the setup, prompts, datasets, grader and results are available in a public repository.

Both approaches finished nearly every task: 255 of 264 procedurally and 258 of 264 declaratively.

Results: less compute, less code

Procedural pipelines stopped no errors before execution, because they have no pre-execution stage. Across the measured builds, procedural runs used 596 executor-seconds, compared with 17 for declarative runs. Databricks attributes the roughly 34x difference to the dry run, which caught structural errors early. In the declarative runs, 55 of 264 builds were stopped before any output.

Code size also differed. Among completed builds, declarative pipelines had a median of 64 lines against 131 for procedural ones, which Databricks reports as about 51% fewer lines. The procedural code also varied more: 19 procedural builds loaded data into memory and processed it row by row, a shortcut that passes on test data but can fail as data grows. None of the declarative builds did this.

Caveats and trade-offs

Databricks lists two caveats. First, a dry run catches only structural mistakes, not silent ones where a pipeline runs cleanly and returns the wrong answer. Silent errors occurred in 73 of 264 procedural builds (27.7%) and 85 of 264 declarative builds (32.2%). The company says the rates were within roughly 15% of each other, too small to count as a real difference, and that logic still needs tests and review.

Second, declarative builds cost more in LLM tokens, about 2.3x. Databricks puts the median build at about $0.50 for declarative versus $0.26 for procedural at Opus 4.8 list rates, as listed in its comparison table. The agent iterates on the dry run until the pipeline is sound, which means more work up front.

What it means for teams

Databricks argues the token cost is a fixed, one-time spend paid while the pipeline is written, whereas wasted cluster compute recurs on every bad run and grows with data size. On that reasoning, the company concludes that the declarative approach comes out ahead at production scale and becomes more cost-effective as pipelines and data grow. These figures come from Databricks' own benchmark of one agent model on 22 tasks, so teams may want to rerun the public setup against their own workloads.