LanceDB Adds GPU Acceleration for Large-Scale Vector Indexing

LanceDB has added GPU acceleration for vector index training, allowing users with Nvidia GPUs or supported Apple Silicon devices to use CUDA or MPS when building indexes. The company says the change can make KMeans training up to 20, 26x faster than CPU-based processing.
- LanceDB v0.3.3 adds CUDA and MPS support for index training.
- GPU acceleration targets KMeans training for IVF indexes.
- LanceDB reports up to 20, 26x faster training than CPU processing.
- The benchmark used 1-million 1536D vectors and IVF_4096.
- PQ training and vector assignment acceleration are planned next.
GPU support targets a costly indexing bottleneck
Vector databases support applications including retrieval-augmented generation, recommendation systems and computer vision. While much of the discussion around them focuses on query latency and recall, LanceDB says the time required to build a vector index can become a major bottleneck as datasets grow.
Building an index is computationally intensive, and the company says the work increases quadratically with the number of vectors or their dimensions. Common techniques such as IVF, or Inverted File index, and PQ, or Product Quantization, divide vectors into clusters. Finding the cluster centroids requires KMeans, an iterative process involving large numbers of vector distance calculations.
LanceDB says GPUs are well suited to this type of computation. Its latest Python package release, LanceDB v0.3.3, backed by Lance v0.8.10, adds GPU acceleration through an accelerator parameter in create_index().
CUDA and MPS are available through the Python package
The implementation supports Nvidia GPUs through CUDA and Apple Silicon GPUs through MPS. Under the hood, LanceDB uses PyTorch to train IVF clusters, then passes the resulting KMeans centroids to its Rust core for index serialization.
The company says this approach provides support for two major developer platforms, Linux and Mac. Users need to install PyTorch, including a CUDA-enabled version where applicable, to use the GPU features.
LanceDB also points to other indexing improvements, including out-of-memory shuffling and batched KMeans training on GPUs. According to the company, these changes allow the system to train indexes on tens of millions of vectors while reducing concerns about CPU or GPU out-of-memory failures.
Tests show faster KMeans training
For its benchmark, LanceDB trained an IVF_4096 index containing 4096 clusters with L2 Euclidean distance. The test used 1-million vectors based on OpenAI Ada2 embeddings with 1536 dimensions, running on both Linux and macOS.
On Linux, the test used a Google Cloud g2-standard-32 instance with 32 logical cores and one Nvidia L4 GPU, running Ubuntu 22.04 with nvidia-driver-525. CPU training took 323 seconds, compared with 12.5 seconds on the GPU.
The macOS test used a 14-inch Macbook Pro with an Apple M2 Max, 64GB of RAM and 30 GPU cores. Training took 397 seconds on the CPU and 21 seconds on the GPU. LanceDB reports that GPU acceleration delivered up to 20, 26x speedups compared with its CPU counterparts.
More index stages are planned for acceleration
The current release accelerates IVF KMeans training, which is only one part of the full index training process. LanceDB says it plans to add GPU acceleration for PQ training and for assigning vectors to their correct centroids. The company expects those additions to produce further improvements in end-to-end index training time.
Because the implementation uses PyTorch, LanceDB says it also has a path toward distributed GPU training and future support for additional hardware accelerators, including TPUs through XLA. CUDA and Apple Silicon GPU support is available through pip install lancedb, giving developers a way to test the feature on compatible hardware now.
LanceDB: GPU-Accelerated Indexing in LanceDB


