Qdrant's Rust cross-encoder library is only 1.1x faster than Python on small models

September 25, 2026Source: Qdrant

Qdrant has rewritten cross-encoder inference in Rust as a library called cross-encode-rs, and its own benchmark shows a modest gain: about 1.1x faster at the median than Python on a small reranking model. The gap is small because the Python libraries and the Rust crate all run on the same C++ engine, onnxruntime.

Key points

Qdrant released cross-encode-rs, a Rust library for cross-encoder inference built on the ort crate.
On the small MiniLM model it was about 1.1x faster at the median and 1.4x faster at p99 than fastembed and sentence-transformers.
On the larger Jina reranker it was 1.3 to 1.5x faster at every percentile except the slowest request (131 ms vs 189 ms and 191 ms at p50).
ONNX session creation time was effectively the same, since all libraries call the same onnxruntime code.
Whole-process timing showed fastembed about 134 ms slower, due to Python startup and imports.

What Qdrant built and why the gain is small

In a blog post by Clelia Bertelli, Qdrant describes how it "oxidized" cross-encoder inference, the Rust community's term for rewriting code in Rust. The result is cross-encode-rs, which uses the ort crate to load and run ONNX models.

A cross-encoder reads a query and a document together and returns one relevance score for the pair. It is typically used for reranking: a fast first stage retrieves candidates, and the cross-encoder re-sorts them. According to Qdrant, this is more accurate than comparing separate embeddings, but also more expensive, because every query-document pair costs a full model run.

Qdrant says the modest speedup does not point to a slow Rust implementation. The Python libraries it compared against, fastembed and sentence-transformers, hand the same work to onnxruntime, so all three spend almost all of their time in the same compiled code.

On the Rust side, the main optimization is batching on two levels. All documents are packed and encoded in one pass rather than one by one, and documents are sorted by tokenized length so that batches contain similar-sized inputs and less computation is wasted on padding. The library also uses the newly released tokenizers v1, which Qdrant says gives a modest speedup over v0.x.

As an example of the scoring, for the query "rust programming", MiniLM scores "cargo builds rust code" at 0.8841 and "iron rusts in wet air" at 0.0035, although both documents contain the same rust token.

Benchmark results against fastembed and sentence-transformers

Qdrant tested the three libraries on the mteb/scidocs-reranking dataset (test split, about 3.98K queries). For each query, positive and negative documents were combined into one array, averaging 30 query-document pairs per query. Two models were used: Xenova/ms-marco-MiniLM-L-6-v2, a small cross-encoder built to run in the browser, and jinaai/jina-reranker-v2-base-multilingual, a larger multilingual reranker. All runs were on a MacBook M4 Max with 48 GB of RAM, using all 14 CPU cores.

MiniLM: cross-encode-rs is about 1.1x faster at the median and 1.4x faster at p99, while the Python libraries are slightly faster on the quickest requests. At p99, a request takes 36 ms with cross-encode-rs, 49 ms with fastembed and 51 ms with sentence-transformers.
Jina: cross-encode-rs is 1.3 to 1.5x faster than both Python libraries at every percentile except the single slowest request. At p50 it takes 131 ms versus 189 ms and 191 ms. The slowest request goes to fastembed at 480 ms, against 581 ms for cross-encode-rs and 585 ms for sentence-transformers.

The two Python libraries performed almost identically, within 5% of each other from minimum to p99 on both models. Qdrant concludes that the runtime is not the whole story: how each library configures and feeds it also counts.

Model load time: no advantage for Rust

Qdrant also measured how long it takes to create the ONNX inference session, which matters for cold starts, autoscaling and serverless deployments. It timed only that step, excluding Python startup, library imports and tokenizer loading, with 11 warmup runs and 41 measured runs per model.

On MiniLM, fastembed was slightly faster, by about 4 ms (12%), creating the session in 33.5 ms on average. On Jina the two were effectively tied, 1% apart on average. Qdrant says the higher cross-encode-rs p99 comes from a single slow run, and that both libraries spend this time in the same onnxruntime call with the same graph optimization level, leaving little room for the language to matter.

When the whole process was timed instead, fastembed took about 134 ms longer than cross-encode-rs on both models. Qdrant attributes that gap to starting the Python interpreter (through uv) and importing the library, and says it is not a criticism of fastembed itself, but overhead a Python service pays on every cold start.

What it means for users

According to Qdrant's numbers, moving reranking from Python to Rust brings little on small models, since the shared onnxruntime engine dominates the time. The benefit grows with a larger model, where the Rust crate was about 1.4x ahead. Teams that care about cold starts may see a gain from avoiding the Python interpreter and imports, though the session creation itself is no faster. These results come from a single machine, a MacBook M4 Max, and two models.