Apple Researchers Train Neural Networks to Speed Up RAG Retrieval

A retrieval problem shaped by real-world queries

Maximum inner product search, or MIPS, is used in recommendation systems, search, and RAG. Given a query vector, a MIPS system searches a database of key vectors for the strongest match. In RAG, the document associated with that match is typically placed before the query so it can help handle the question.

Exhaustively comparing a query with every key becomes increasingly expensive as databases reach millions of keys and vector dimensions reach thousands. Existing approximate methods use techniques such as clustering and quantization to reduce the work, but Apple says these methods can overlook the difference between queries and keys. In RAG, queries are often short, user-generated questions, while keys represent longer, curated pieces of text.

Apple also points to production systems with stable and well-characterized query distributions. Its researchers argue that methods designed around the queries a system is likely to receive can use computing resources more efficiently than generic, query-agnostic approaches.

Learning how to route queries

In a paper presented at ICML 2026, Apple researchers describe training small neural networks once, offline, for a fixed database of keys. These networks learn how typical queries are matched, creating an upfront cost in exchange for faster and more accurate retrieval during inference, according to the research team.

The approach is based on the mathematical structure of MIPS. The function that maps a query to its largest inner product with any key is the support function of the key set. Apple says this function is convex, making it suitable for approximation with input convex neural networks, or ICNNs. Its gradient at a query returns the best-matching key.

The researchers train models using sampled user queries and ground-truth outputs generated through exhaustive search. The resulting supervised data captures the MIPS solution landscape for the query distribution that matters to the system.

SupportNet and KeyNet take different roles

Apple describes two specialized model types. SupportNet learns the support function directly and can estimate one support function for each data cluster. Those estimates can improve routing by measuring how well the best key in a cluster matches a query, rather than relying only on the distance between the query and a cluster centroid.

Centroid routing can miss a strong match when a relevant key is far from its cluster’s centroid but close to the query. SupportNet is designed to account for the best key inside each cluster when selecting which clusters to probe.

KeyNet models instead learn to output the best-matching key vector directly. A consistency loss based on Euler’s theorem for homogeneous functions keeps the predictions connected to the underlying support function, even though KeyNet does not explicitly enforce convexity.

Why the approach matters for RAG systems

The research offers a way to use known query patterns when reducing the cost of large-scale retrieval. That could be useful for RAG systems, where selecting the right document affects the information available to answer a question, and where short user questions can differ substantially from the stored documents used as keys.

Apple reports an evaluation using Natural Questions, partitioned into 128 clusters, with roughly 2.6M unique passages in its setup. The researchers trained SupportNet models in four sizes, labelled XS, S, M, and L, corresponding to 1%, 5%, 10%, and 20% of the database size. The source describes retrieval performance in terms of recall and cost, but the provided announcement does not include the resulting figures.