deepset adds agent cost optimization and custom models to Haystack

Cost visibility and testing inside the platform

deepset presented the offering at NVIDIA GTC Berlin. The company argues that controlling AI spend means looking beyond individual models to how they are used within agent systems. It says it helps organizations optimize model choice and the harness together, testing changes against cost and quality requirements while they keep control over models, data and infrastructure.

Haystack Enterprise Platform now shows AI spend by model, agent, service and team. The data appears both in centralized reporting and alongside production traces. According to deepset, teams can find expensive or unreliable steps, compare models and harness configurations, and evaluate systematically whether a change lowers cost while still meeting the quality the task requires.

The company says the platform covers the full loop of building, running, optimizing and operating agents, so improvements can be identified and applied within the same environment. Upcoming enhancements are meant to automate this loop further by continuously recommending ways to reduce cost and improve quality of production agents, with human approval before deployment.

Custom models for repeated tasks

The second part of the announcement concerns specialized models. deepset says industry-tuned decision models and small language models can handle repeated tasks such as classification, routing and verification more efficiently. That reduces reliance on frontier LLMs, lowers costs and allows deployment on locally controlled infrastructure.

Its custom AI model services target government, defense, energy and regulated industries. Forward Deployed Engineers assess customer requirements, select, adapt or post-train models, and evaluate them against quality and cost objectives before integrating them into production agent workflows. According to deepset, customers own the models developed for them. Haystack's model-agnostic orchestration connects these models with larger LLMs, enterprise data and tools, so components can be changed as new innovations or requirements emerge.

CEO Milos Rusic framed the approach this way: "AI economics will not be solved by choosing the cheapest model or defaulting to whichever model leads the market this week." He added that the goal is to complete the task at the lowest cost that meets quality and security requirements, which sometimes means a smaller, specialized model and sometimes improving the harness around a larger one.

Deployment options and NVIDIA ties

Organizations can deploy Haystack Enterprise Platform in public cloud, sovereign cloud environments such as STACKIT, or self-hosted infrastructure. Shared permissions, policies and usage controls govern how agents access models, tools and data. deepset says this lets teams improve AI economics within their security and regulatory requirements without tying applications to a single model provider or deployment environment.

The platform uses NVIDIA Triton Inference Server for shared, GPU-accelerated embedding and reranking. Through the custom model services, deepset engineers can also assess and tailor NVIDIA Nemotron models where they suit a customer's tasks. At GTC Berlin, deepset will join NVIDIA for sessions on Trustworthy AI Platforms and Secure, Self-Improving Agents for On-Prem AI, and can be found at booth #3071.

What it means for organizations

The announcement builds on deepset's work with the European Commission, whose self-hosted AI@EC platform supports governed AI applications, Germany's Federal Ministry of Research, Technology and Space (BMFTR), which uses an AI assistant for project-funding guidance, and Bosch, whose DRIVE Companion supports its SAP migration teams.

For organizations running agents in production, particularly in regulated sectors, the offering combines spend reporting, systematic testing and the option of owning tailored smaller models. The source gives no pricing, availability dates for the upcoming automation features or performance figures, so the size of any savings remains to be shown in practice.