turbopuffer adds native embedding to cut semantic search latency
turbopuffer has launched native embedding, letting customers hand the embedding step to the search engine instead of running their own pipeline. According to the company, running embedding inside the system allows optimizations that can shave hundreds of milliseconds off a query.
- turbopuffer launched native embedding, so it can generate embeddings for writes and queries instead of customers doing it.
- The query planner can overlap embedding with work like reading namespace metadata, which turbopuffer says can save hundreds of milliseconds per query.
- During beta, Linear cut 150ms from its embedding pipeline and Readwise reduced median embedding latency by 8x, according to turbopuffer.
- Customers can use native embedding for writes only, queries only, or not at all.
- Planned additions include background re-embedding of namespaces, native reranking, better parsing and chunking, and search agents.
From bring your own embeddings to native embedding
turbopuffer has historically been BYOE, short for bring your own embeddings. The company says this let it focus on the scale and performance of semantic search, while customers kept the flexibility to fine-tune embeddings for specific domains such as finance, law and coding.
Over time, turbopuffer says, many customers came to see building an embedding pipeline less as a chance to optimize and more as a chore, involving "weird APIs, batching, retries, and timeouts." Native embedding, now generally available, is meant to take that work off their hands.
Engineer Ben Linsay, who wrote the announcement, says convenience was not the only goal. Because the embedding call now sits inside turbopuffer, the company can apply whole-system optimizations that were not possible before.
How overlapping work reduces latency
When embedding happens outside turbopuffer, it runs on a different computer and is invisible to the query planner. It is also serialized with query execution: a document vector cannot be scored until a query vector exists, and the client mostly waits in the meantime.
With native embedding, the query planner can see the embedding step and run it in parallel with the initial phase of query execution. According to turbopuffer, reading namespace metadata from the index does not depend on the query vector, so it can happen while the GPU is busy producing the embedding.
The scan still has to wait for the vector, because turbopuffer needs it to pick clusters and score documents. The company describes its trace as simplified and says it keeps looking for ways to push that dependency further down the pipeline.
Two customers that used native embedding during the beta have reported gains, according to turbopuffer. Linear shaved 150ms off its embedding pipeline, and Readwise reduced its median embedding latency by 8x. The company says it is observing similar gains now that the feature is generally available.
Tradeoffs and what comes next
turbopuffer notes that native embedding fits the most common workload, where every write and every read needs a new embedding. It is the wrong tradeoff for some cases. If a query embedding can be reused over time or across namespaces, the company advises against re-embedding on every read, since avoiding a call entirely beats optimizing it.
Customers can therefore move embedding into turbopuffer only for writes, only for queries, or not at all. Future versions are planned to cache repeated calls to embed the same document automatically and pass the savings on to customers.
The company also outlined its roadmap. It is working on database improvements for asynchronously re-embedding an entire namespace in the background. From there, it suggests, it is a small step to running evaluations on new models and automatically migrating to the best performer. Also listed as coming soon are native reranking, better document parsing and chunking, and search agents.
turbopuffer describes itself as a search engine that hosts 1T+ documents, handles 10M+ writes/s and serves 25k+ queries/s. For teams that currently maintain their own embedding code, the announcement points to less pipeline work and lower query latency, with the option to keep embedding outside the system where that makes more sense.
turbopuffer: Why moving embedding inside turbopuffer drops search latency
