Databricks adds Predictive AI Functions for forecasting and anomaly detection in SQL

Five functions for common analytics questions

According to Databricks, data teams keep getting the same questions: what sales will be next quarter, why a metric moved, and whether recent data falls outside the norm. Answering them usually means passing the work to a data science team that builds models and manages infrastructure. Databricks says this creates a bottleneck in which business users wait in line and many questions are never asked.

The launch adds four SQL-native functions and improves the accuracy of ai_forecast:

• ai_forecast: projects a time series into the future
• ai_top_drivers: finds which segments most drove a change in a metric
• ai_detect_anomalies: flags outlier rows in temporal data
• ai_predict_value: predicts continuous values (regression)
• ai_predict_class: predicts categorical values (classification)

The functions can be called directly in SQL, invoked in Genie with plain-language questions, or embedded in dashboards. Databricks maintains the underlying models centrally, so teams do not have to build or manage infrastructure.

A new ai_forecast with holiday and covariate support

The upgraded ai_forecast is powered by a transformer-based time series foundation model. Databricks says it was pretrained on a large, diverse collection of time series, in the way a large language model is trained on text. Because it already recognizes patterns such as seasonality and trends, the company says it produces accurate forecasts with no tuning required.

In a dashboard example using the NYC taxi trips dataset, Databricks projected monthly ride volumes through the end of 2027 and showed the expected seasonal patterns in taxi demand.

The new version also lets teams add business-specific factors. Built-in holiday support can account for local events such as Christmas dips and school-holiday spikes. Teams can also supply external covariates, such as pricing changes or marketing spend, so that forecasts better reflect their business.

Anomalies, drivers and predictions in SQL and Genie

The four new functions are aimed at ad-hoc questions. ai_top_drivers explains why a metric moved, ai_detect_anomalies flags values that look off, and ai_predict_value and ai_predict_class fill in missing values by estimating a number or assigning a category.

Analysts can call them in the SQL editor or a notebook and run them across hundreds of millions of rows, according to Databricks. In one example, daily taxi revenue is aggregated into a time series and passed to ai_detect_anomalies, which fits a model automatically and flags unusual days. In another, ai_top_drivers ranked time-of-day and weekday/weekend segments by their impact on the change in NYC taxi fare revenue between January and February 2016. Weekday midday hours drove the largest gain, while the weekday evening rush was the biggest drag.

In Genie, ai_top_drivers is available today. ai_forecast, ai_detect_anomalies, ai_predict_value and ai_predict_class are coming soon. Users will be able to ask questions such as why revenue dropped last quarter or which customers are likely to churn, and Genie will call the appropriate function. Databricks says these tailored methods let Genie answer questions that a general-purpose LLM cannot reliably handle on its own.

Benchmark claims and what it means for teams

Databricks used fev-bench to compare the new ai_forecast with functions from what it calls a leading cloud data warehouse. The metric was task win rate, defined as the percentage of tasks, measured by scaled quantile loss, on which one system has a lower probabilistic error than the other. The company says ai_forecast outperforms other cloud data warehouse providers as well as the previous version of ai_forecast. The source text does not include the underlying figures. These are the company's own results.

For organizations on Databricks, the practical change is that forecasting, driver analysis and anomaly detection become available to people who work in dashboards, Genie and SQL, without a separate modeling project. Databricks says this shortens development time compared with building custom models. The functions are still in Beta.