Qdrant 1.20 adds BM25 on full-text indexes and TurboQuant datatypes

BM25 scoring without sparse vectors

Qdrant supports full-text search, which finds results by the exact words they contain. Until now, ranking those results with BM25, a commonly used ranking function for keyword search, meant converting text into sparse vectors. According to Qdrant, that approach has drawbacks: every document and query needs an inference step, and users must track the average document length themselves. Because that value is fixed in every stored vector, it cannot adapt as data changes, and updating it means converting all documents again.

Version 1.20 adds BM25 scoring directly on a full-text index. Qdrant derives all BM25 statistics from the data, including the average document length. Text already in the payload becomes searchable as soon as it is indexed.

To use it, a user creates a full-text index on a field with the bm25 scoring type, then sends a text query and names the indexed field with using. The BM25 parameters k and b can be set per query, so tuning does not require rebuilding the index. A text query works anywhere in the Query API, including hybrid search with dense vectors or as a rescoring step for points retrieved by another query.

BM25 on sparse vectors stays supported. Users whose text is already in the payload can switch without re-uploading data: create a full-text index with scoring enabled, configure the same text processing as the sparse BM25 model, and replace sparse vector queries with text queries.

Code tokenizer and faster nested filters

The default word tokenizer splits text on spaces and punctuation, which Qdrant says works poorly for source code. It indexes getUserName as a single token, so a search for user name or get_user_name does not find it. The new code tokenizer splits identifiers on underscores and case changes, including acronyms, so getUserName, get_user_name and user name all match each other. It also indexes the full identifier, so hashmap still finds HashMap. Combined with BM25 scoring, this allows ranked code search without embeddings.

Nested filters, which find points where a single array element meets all conditions, also get an upgrade. Previously, indexes on fields such as offers[].shop and offers[].price could only narrow down candidates, and Qdrant then read the payload of each remaining candidate, usually from disk. Every payload index type now has a nested parameter that creates a per-element index. When all conditions in a nested filter use fields with a per-element index on the same array, Qdrant answers the filter from the indexes alone.

The search API does not change, so existing nested filters get faster once the indexes are created. In a benchmark with three nested filters on 100,000 points, Qdrant reports per-element indexes were 7 to 38 times faster than regular indexes with payload reads, even with all payloads in memory.

turbo16 and turbo8 datatypes

The release adds two TurboQuant datatypes. Previously, Qdrant offered turbo4 (4 bits, introduced in version 1.19, with a significant negative effect on recall on some datasets), float16 and uint8.

turbo16 uses 16 bits per dimension like float16 but applies the TurboQuant rotation first, then stores each vector on a single 16-bit integer grid. Qdrant says it halves the storage of float32 vectors with no loss in accuracy. Across 13 datasets, turbo16 had 7.5 times lower reconstruction error than float16. In search benchmarks on 12 datasets, its recall@10 with exact search matched float32 within 0.001 on every dataset, while float16 dropped to 0.995 on the ada-002 and CLIP datasets. For HNSW search, median latency was 0.81 times that of float16 on x86 and 0.83 times on ARM.

turbo8 uses 1 byte per dimension and quantizes regular float vectors automatically, unlike uint8. Qdrant says its recall comes close to float16 at half the size. Across 11 datasets, searches were 1.1 to 2.4 times faster than float16. Against float32 vectors with scalar quantization and rescoring, turbo8 had higher recall on 10 of 11 datasets and tied on the other, at similar latency, while storing about a fifth of the bytes.

The 8-bit encoding is also available as TurboQuant quantization by setting bits to bits8. It compresses vectors about four times, compared to eight times for bits4, which remains the default.

What it means for users

Teams that run keyword search on payload text can drop the embedding step and keep BM25 statistics current automatically. Those with nested payload filters can add per-element indexes without changing queries. Users on scalar quantization are advised by Qdrant to benchmark bits8 as an alternative, since it offers the same four-times compression and, in the company's tests, higher recall.