Databricks details a real-time recommendation stack for fashion e-commerce

One platform instead of fragmented ML infrastructure

The blog post argues that building a production-grade recommendation system is one of the hardest ML engineering challenges. It requires real-time data ingestion, complex feature engineering, several models working together, and serving infrastructure that responds in milliseconds, while inventory, location and business rules stay in sync.

Databricks says its design replaces fragmented ML infrastructure with a single platform governed by Unity Catalog. The company claims this removes glue code, speeds up iteration cycles and gives end-to-end lineage from raw clickstream events to production predictions. It also cites industry benchmarks suggesting that effective personalization can lift conversion rates by 10 to 30% and increase average order value significantly.

The system combines pre-computed batch recommendations with a real-time scoring path. Databricks AI Search handles candidate retrieval, Lakebase provides online feature serving, and Model Serving delivers low-latency inference.

From clickstream to medallion layers

The platform ingests approximately 1,000 events per second, including product views, searches, add-to-cart actions, purchases and session metadata. Lakeflow Connect's Zerobus Ingest lands these events directly into Unity Catalog Delta tables without a self-managed message broker. According to Databricks, Zerobus accepts data from any standard Kafka producer client in Java, Python or Go through a configuration change that points the bootstrap server at the Zerobus endpoint. Teams that already run Kafka can instead use Structured Streaming with Declarative Pipelines, with the same downstream architecture.

Data then flows through a medallion architecture:

• Bronze: raw, append-only event streams plus reference data, covering user signals (behavioral, transactional, preference, demographic), item signals (catalog, performance, visual, freshness) and contextual signals (temporal, geographic, environmental, session).
• Silver: sessionized user behavior, cleaned product features, aggregated engagement signals such as 7-day view counts and trending scores, and user-product interaction matrices.
• Gold: user and item feature vectors, labeled training datasets and pre-computed recommendation lists.

Behavioral aggregates update daily through scheduled Databricks Workflows, while the full product catalog syncs weekly. Embeddings for users and items are recomputed daily. The Databricks Feature Store manages offline features for training and online features for serving, so the same feature definitions are available at inference time through Lakebase online tables. Unity Catalog adds fine-grained access control, which the company says keeps PII protected while aggregated features flow to model training.

Path A: pre-computed recommendations

The serving layer has two paths. Path A covers the high-volume, predictable surfaces: homepage carousels, category page rankings, email campaigns and push notifications. For these, the user and the surface type are known in advance, so results can be computed ahead of time.

A nightly batch job orchestrated by Databricks Workflows runs the full 3-stage funnel offline for every active user. It retrieves the latest user embeddings, runs batch ANN queries against the AI Search item index to generate candidates, scores them with a LightGBM model using Gold layer features, and applies business rules such as inventory scoring, delivery proximity, diversity and promotional boosting. The result is a top-N ranked list per user, typically 50 to 100 items per surface, written to Lakebase online tables keyed by user ID and surface type.

At request time the app does a simple key-value lookup of user ID plus surface. There is no model inference, vector search or feature assembly. Databricks labels the latency for this path as under two-digit milliseconds. Because the job runs nightly, results reflect the previous day's signals and inventory. The company says that is more than sufficient for most surfaces, since long-term preferences and brand affinities evolve over days, and new products with initial embeddings appear in recommendations within 24 hours.

Path B: session-aware scoring at request time

Path B is used when the context exists only at request time, such as "Similar items" on a product detail page, "Complete the look" suggestions, or search results that are re-ranked as the user browses. The app sends current session signals, including items viewed in the last few minutes, active search queries, cart contents and dwell-time patterns, directly as the request payload to the Model Serving endpoint over REST. Databricks stresses that this bypasses lakehouse storage during inference, so real-time context is available without ingestion latency.

The endpoint runs the full 3-stage funnel synchronously in one request-response cycle. In stage 1, it turns session signals into a query embedding. This is either the embedding of the product being viewed or a blended user embedding that fuses the long-term profile from Lakebase with in-session behavior, weighting recent signals more heavily. Databricks AI Search then performs hybrid retrieval, combining ANN similarity search with hard metadata filters for category eligibility, regional inventory and minimum stock thresholds, so out-of-stock or ineligible items never enter the candidate set.

What it means for retail teams

The post is a reference design rather than a product launch, and it gives no pricing or availability details. For retail data teams, it shows how Databricks proposes to combine its ingestion, feature, vector search and serving components for personalization. The split between a cheap key-value path for predictable surfaces and a synchronous scoring path for session-driven ones is the central design choice. Teams already on Databricks or Kafka can see which parts of an existing stack the company suggests replacing or reusing.