Artificial intelligence news.
Straight from the original announcements.

Lance File 2.1 targets smaller storage and faster random access

November 19, 2026, 08:16By the AI News Bites Editorial TeamSource: LanceDB
Lance File 2.1 targets smaller storage and faster random access
Image: LanceDB

Lance File 2.1 has entered beta with a redesigned encoding system intended to reduce file size and improve access to individual values. Lance says the update builds on the 2.0 format while addressing lessons from community use and production deployments.

Key points
  • Lance File 2.1 has entered beta.
  • The format separates structural encoding from compressive encoding.
  • Lance says 2.1 can access any value in one or two IOPs.
  • The design keeps the 2.0 structure of pages, column descriptors and a file footer.
  • Lance says 2.0 sometimes beat Parquet scans despite files being 2-3 times bigger.

A smaller, more flexible format

In a blog post, Lance engineer Weston Pace described 2.1 as a response to several months of work with the community and production users. The company’s 2.0 format became Lance’s default file format last fall, after a beta period.

Pace said the 2.0 design successfully removed row groups without sacrificing parallelism or performance. Lance also developed scheduling intended to keep its I/O queue full while staying within a parallelism budget. According to Lance, this approach helped the format match or exceed Parquet scan performance in some cases, even when the Lance file was 2-3 times bigger.

The underlying file structure is staying largely intact. Lance says pages, column descriptors and a file footer provide a simple and flexible foundation. The major changes in 2.1 concern encodings, where the team has reworked much of the design.

Separating structure from compression

Lance File 2.1 separates encoding into two concepts: structural encoding and compressive encoding. Structural encoding determines how an array is divided into buffers, while compressive encoding determines how those buffers are compressed.

The distinction is important for a format designed to handle multimodal data such as embeddings, tensors, audio, video and images. Lance says these data types are often already compressed, which made compression a lower priority in 2.0. However, compression still matters for metadata, prefiltering and text-heavy datasets such as Common Crawl, Fineweb and Github Code.

The new design also accounts for the tradeoff between compression and random access. Lance says transparent techniques such as bitpacking can support random access, while opaque methods such as delta encoding may not. The structural encoding chosen for a value helps determine which compressive encodings can be used.

Aiming for one or two IOPs per value

One of Lance’s central goals for 2.1 is what Pace calls the “1-2 IOP challenge”. For fixed-width data types, the target is access in a single I/O operation, or IOP. Variable-width types should require two.

In Lance 2.0, a single value in a complex type such as a list of strings could require six IOPs. The relevant information could include validity buffers, list and string offsets, and string values. Lance says 2.1 combines structural encodings with a repetition index to reduce that cost.

According to the company, the result is access to any value in one or two IOPs, regardless of the data type’s complexity or the number of validity layers. The 2.1 format is entering beta, so the announcement describes a design and direction rather than a final release schedule.

Related news

Lance introduces community governance for its open format
For developersNovember 19, 2026, 08:16

Lance introduces community governance for its open format

Lance has introduced a formal governance structure for its open format community, defining contribution paths, leadership roles and consensus-based decisions. The…

LanceDB Adds GPU Acceleration for Large-Scale Vector Indexing
For developersNovember 19, 2026, 08:16

LanceDB Adds GPU Acceleration for Large-Scale Vector Indexing

LanceDB has added GPU acceleration for vector index training, allowing users with Nvidia GPUs or supported Apple Silicon devices to use CUDA or MPS when building…

Lance v2 targets AI workloads with a flexible columnar format
For developersNovember 19, 2026, 08:16

Lance v2 targets AI workloads with a flexible columnar format

LanceDB has outlined Lance v2, a new columnar container format designed to address limitations the company sees in Parquet and Lance v1 for modern AI and machine…