Databricks makes Real-Time Mode generally available for Spark Declarative Pipelines

What Real-Time Mode does

Databricks describes RTM as an evolution of the Spark Structured Streaming engine. It reaches millisecond-range end-to-end latencies by streaming data continuously, running all query stages at the same time and passing data between stages as soon as it is produced, instead of working in micro-batches.

The company introduced RTM last year for Spark Structured Streaming. It says that version allowed customers to process streaming data in milliseconds, up to 100x faster than traditional micro-batch architecture, and spared them from maintaining a specialized engine like Flink next to Spark. Databricks says this removed duplicate codebases, fragmented governance models and operational overhead.

The new announcement brings that capability to SDP. Databricks says SDP automatically handles dependency resolution, incremental processing, orchestration, failure recovery and data quality enforcement, and that RTM pipelines keep all of those benefits.

Benchmarks and the feature computation use case

Databricks says it benchmarked RTM against Apache Flink on actual customer workloads performing feature computation, and that RTM is up to 92% faster than specialized engines like Flink. The datasets and queries are referenced in a GitHub repository linked from the post.

Feature computation is the use case the company highlights. ML models for fraud detection or personalization depend on fresh signals, for example a user's total transaction amount over the last 10 minutes. Databricks says RTM on SDP is designed to compute such rolling-window aggregations and land them in an online store within milliseconds. According to the company, the Databricks Feature Store uses this architecture to reach an end-to-end p99 latency of 200ms from Kafka to the online feature store.

Other example workloads named in the announcement include live personalization, real-time recommendation systems and real-time alerting.

Setup and custom sinks

Enabling RTM takes two steps, according to the post. First, the pipeline mode must be continuous, with spark.databricks.streaming.realTimeMode.enabled set to true. Second, users define a real-time update flow: they create the output target with dp.create_sink(), then use the @dp.update_flow decorator with pipelines.trigger set to "RealTime" and the target pointing to the sink.

Pipelines can write to any custom sink through a ForEach sink, including Lakebase, Redis or any other OLTP store. The writer implements open, process and close, is registered as a foreach_sink and is wired to an update_flow. Databricks says features then land in the operational store within milliseconds of being produced, with no offline-to-online sync in between.

On SDP serverless, the company says there is no compute to choose and no cluster to size, and customers pay only for the cores their query uses. RTM flows also get versionless upgrades and low-downtime maintenance.

What comes next and why it matters

Databricks lists two planned improvements for RTM on serverless. The first is lower-cost operation by bin-packing multiple RTM flows into a single pipeline, with the system auto-tuning source and shuffle partitions based on workload. The second is expanded sink support, including a native Lakebase sink for SDP with built-in batching, automatic retries and UC-managed authentication.

For data teams already using SDP, the company's pitch is consolidation: one framework, one observability stack, one billing model and no fragmentation across different APIs. If the performance claims hold up on a team's own workloads, latency-sensitive pipelines such as fraud scoring and feature serving could run on the same APIs as the rest of their ETL, rather than on a separate engine.