Artificial intelligence news.
Straight from the original announcements.

LanceDB proposes multi-base paths for distributed Lance datasets

November 16, 2026, 18:17By the AI News Bites Editorial TeamSource: LanceDB
LanceDB proposes multi-base paths for distributed Lance datasets
Image: LanceDB

LanceDB has introduced Lance’s multi-base path model, a design aimed at distributing one dataset across multiple storage locations without giving up the portability of relative paths. The company says the work grew from a production requirement from Uber’s AI infrastructure team.

Key points
  • LanceDB introduced Lance’s multi-base path model.
  • The design grew from an Uber requirement to distribute one dataset across multiple S3 buckets.
  • Lance uses relative paths and a defined directory structure.
  • LanceDB links the design to training, agentic search and shallow clones.

A storage problem at petabyte scale

In a blog post dated November 16, 2026, LanceDB explains that table formats do more than store files. They define how data is assembled, with metadata linking the files that make up a table. The way those links represent paths can affect portability, migration and operational complexity when datasets reach petabyte scale.

Uber needed to distribute a single Lance dataset across multiple S3 buckets to increase throughput, while retaining the advantages of relative paths. LanceDB presents the multi-base layout as its response to that requirement. The post also highlights use cases involving multimodal data distributed across storage buckets, more than 1K concurrent readers for training and agentic search, and zero source data copies for shallow clones.

Why existing path models create trade-offs

LanceDB traces the design debate through Apache Iceberg and Delta Lake. Iceberg originally used absolute paths, which can identify a file without additional context and allow files to live outside a table root. However, relocating a table can require rewriting the many metadata and manifest files that contain those paths. The post says the Iceberg community began aligning in late 2025 on a v4 design for relative paths intended to support relocation without metadata rewrites.

Delta Lake began with relative paths in its protocol specification, published in September 2019. It later allowed absolute paths in August 2021, helping support capabilities such as shallow clone, which was introduced in December 2022. According to LanceDB, this hybrid approach adds flexibility but makes path resolution and operations such as credentials vending more complex when files span multiple locations.

Lance prioritizes predictable, relative paths

LanceDB says Lance was designed around two properties: predictable file placement and strict portability. Its format defines directories including data/, _versions/, _transactions/, _deletions/ and _indices/, with fixed paths used where possible to make resolution deterministic and reduce manifest size.

The company also says every path in Lance is relative. As a result, copying a dataset directory to another location does not require metadata updates. LanceDB describes that behavior as important for AI teams that move data between local experiments, cloud storage, distributed training and analysis environments.

The multi-base model is presented as the next step: supporting a dataset spread across locations while preserving the portability and predictable structure that LanceDB considers central to Lance. For users operating large AI datasets, the proposal targets simpler relocation and storage distribution without requiring the format to rely on unrestricted absolute file references.

Related news

LanceDB Adds GPU Acceleration for Large-Scale Vector Indexing
For developersNovember 19, 2026, 08:16

LanceDB Adds GPU Acceleration for Large-Scale Vector Indexing

LanceDB has added GPU acceleration for vector index training, allowing users with Nvidia GPUs or supported Apple Silicon devices to use CUDA or MPS when building…

Together AI launches free storage caching for GPU clusters
Products and toolsNovember 19, 2026

Together AI launches free storage caching for GPU clusters

Together AI has launched Together Storage Accelerator, a distributed caching layer for its GPU Clusters that serves repeated data reads from local NVMe storage. The…

Fivetran explains how Apache Polaris catalogs managed Iceberg tables
Products and toolsNovember 18, 2026, 05:00

Fivetran explains how Apache Polaris catalogs managed Iceberg tables

Fivetran has published guidance for using Apache Polaris as the catalog behind its Managed Data Lake Service. The service uses Polaris to provide a REST-based interface…