Lance File 2.1 targets smaller storage and faster random access

Lance File 2.1 has entered beta with a redesigned encoding system intended to reduce file size and improve access to individual values. Lance says the update builds on the 2.0 format while addressing lessons from community use and production deployments.
- Lance File 2.1 has entered beta.
- The format separates structural encoding from compressive encoding.
- Lance says 2.1 can access any value in one or two IOPs.
- The design keeps the 2.0 structure of pages, column descriptors and a file footer.
- Lance says 2.0 sometimes beat Parquet scans despite files being 2-3 times bigger.
A smaller, more flexible format
In a blog post, Lance engineer Weston Pace described 2.1 as a response to several months of work with the community and production users. The company’s 2.0 format became Lance’s default file format last fall, after a beta period.
Pace said the 2.0 design successfully removed row groups without sacrificing parallelism or performance. Lance also developed scheduling intended to keep its I/O queue full while staying within a parallelism budget. According to Lance, this approach helped the format match or exceed Parquet scan performance in some cases, even when the Lance file was 2-3 times bigger.
The underlying file structure is staying largely intact. Lance says pages, column descriptors and a file footer provide a simple and flexible foundation. The major changes in 2.1 concern encodings, where the team has reworked much of the design.
Separating structure from compression
Lance File 2.1 separates encoding into two concepts: structural encoding and compressive encoding. Structural encoding determines how an array is divided into buffers, while compressive encoding determines how those buffers are compressed.
The distinction is important for a format designed to handle multimodal data such as embeddings, tensors, audio, video and images. Lance says these data types are often already compressed, which made compression a lower priority in 2.0. However, compression still matters for metadata, prefiltering and text-heavy datasets such as Common Crawl, Fineweb and Github Code.
The new design also accounts for the tradeoff between compression and random access. Lance says transparent techniques such as bitpacking can support random access, while opaque methods such as delta encoding may not. The structural encoding chosen for a value helps determine which compressive encodings can be used.
Aiming for one or two IOPs per value
One of Lance’s central goals for 2.1 is what Pace calls the “1-2 IOP challenge”. For fixed-width data types, the target is access in a single I/O operation, or IOP. Variable-width types should require two.
In Lance 2.0, a single value in a complex type such as a list of strings could require six IOPs. The relevant information could include validity buffers, list and string offsets, and string values. Lance says 2.1 combines structural encodings with a repetition index to reduce that cost.
According to the company, the result is access to any value in one or two IOPs, regardless of the data type’s complexity or the number of validity layers. The 2.1 format is entering beta, so the announcement describes a design and direction rather than a final release schedule.
LanceDB: Lance File 2.1: Smaller and Simpler


