LanceDB Highlights Robot Training, DataFusion Joins and LLM Pipelines

LanceDB says LeRobot v0.7.0 can train on robot datasets stored in Lance tables without downloading them, while its latest work also adds ASOF JOIN to Apache DataFusion and demonstrates LLM pretraining from a single table.
- LeRobot v0.7.0 trains on Lance tables without downloading datasets.
- Original MP4 files remain as blobs and frames are read in place.
- Apache DataFusion now includes ASOF JOIN from LanceDB engineers.
- A walkthrough stores 2.43B GPT-2 tokens in one LanceDB table.
- Lance and LanceDB releases add indexes, materialized views and security changes.
LeRobot trains directly from Lance tables
The Hub now holds almost 300 TB of open robot data in LeRobot format, according to LanceDB. With LeRobot v0.7.0, users can train on that data without downloading it. The release was built with the LanceDB team and reads datasets stored as Lance tables, while keeping the original MP4 files as blobs.
That design means the videos are not re-encoded, and frames are read in place. LanceDB says the approach provides a true global shuffle at local-disk speed. The existing LeRobotDataset API remains unchanged. Users can convert an existing dataset with lerobot-lance-convert and browse it in lerobot-dataset-viz.
DataFusion gains ASOF JOIN for time-aligned data
LanceDB engineers have added ASOF JOIN to Apache DataFusion, the query engine behind LanceDB's SQL. The operation is designed for data recorded at different frequencies, such as camera frames and robot states.
For each frame, an ASOF join finds the latest state at or before the frame's timestamp. Because the feature is part of Apache DataFusion, LanceDB says every DataFusion user gets it. The contribution follows upstream work on LeRobot, Apache OpenDAL, CocoIndex, and ICU4X.
One LanceDB table supports an LLM pretraining walkthrough
LanceDB's walkthrough addresses a common pipeline pattern in which each pretraining stage writes a new dataset. In that setup, changing a block length or filter can require a rebuild. The example instead keeps raw text, deduplication flags, and 2.43B GPT-2 tokens in one LanceDB table.
Sequences are packed inside StreamingDataset at read time. LanceDB says raw text becomes training-ready in 11 minutes, while GPT-2 124M trains at 3.18M tokens/s and 34.5% MFU in the walkthrough. The approach is aimed at making data preparation changes without recreating every intermediate dataset.
Storage and ecosystem updates broaden the platform
The newsletter also lists releases across Lance, LanceDB, lance-namespace-impls, and lance-spark. Lance v13.0.0 through v14.0.2 adds BM25F cross-field full-text search, an experimental MinHash LSH scalar index, and changes to Btree scalar indexes. LanceDB v0.40.0 through v0.41.0 adds SQL-defined materialized views, new scalar index builders, JSON field FTS indexes, and broader blob handling in Node.js.
One security-related change means embedding configuration stored in table metadata is no longer trusted by default. SentenceTransformerEmbeddings.trust_remote_code defaults to False, and restored remote code or custom endpoints require explicit approval.
For users, the updates connect robot learning, analytical joins, dataset preparation, and indexing within LanceDB's broader open source ecosystem. LanceDB's announcement presents the changes as practical ways to reduce data movement, avoid repeated rebuilds, and improve work with large datasets, while DataFusion users can access the new time-based join independently of LanceDB.


