Datadog adds Fireworks AI as a model provider for Agent Observability

Bring your own Fireworks AI key

The integration targets teams that build agents on open models hosted on Fireworks AI, such as GPT-OSS, Kimi, GLM, Qwen and MiniMax. Datadog says these teams need a reliable way to measure whether their LLM applications and agents give accurate, safe and helpful answers efficiently.

Customers bring their own Fireworks AI API key. After the account is connected once, every model it can access becomes available wherever Agent Observability lets users choose a model provider. Datadog says this gives end-to-end visibility into agents built on Fireworks AI, from traces and token usage to quality evaluations.

LLM-as-a-judge evaluations on Fireworks AI models

LLM-as-a-judge evaluators use a model to score the inputs and outputs of an LLM application. Datadog says this helps catch issues such as hallucinations, off-topic answers or negative user sentiment at scale.

With Fireworks AI models acting as the judge, teams can match the model to the task. Datadog gives three examples:

• a larger model such as gpt-oss-120b for nuanced quality checks
• a low-latency router endpoint for high-volume scoring
• a customer's own fine-tuned model for domain-specific criteria

Because the evaluations run on the customer's Fireworks AI account, model usage follows the billing and governance controls already set up there, and teams choose the trade-off between cost and quality for each evaluator. Datadog also says there is no need to build a separate pipeline to export traces, call a model and write scores back.

Datadog runs the evaluator against production traces with the selected model, scoring qualities such as faithfulness, topic relevance or sentiment. Results appear next to each span. Users can track them over time, filter traces by score and set alerts on regressions.

Testing prompts in the Playground

The Agent Observability Playground lets teams iterate on prompts before they reach production. Datadog describes an example in which an online travel agency runs a support assistant and wants to know whether an open model can answer refund and cancellation questions accurately and quickly enough for live chat.

In the Playground, a user can run a system prompt and a real customer question against gpt-oss-120b and see the response alongside its token count and latency. Other Fireworks AI models can then be swapped in, including low-latency router endpoints like Nexus Auto, to weigh answer quality against speed and cost. The full trace of any run can be opened to see what the model received and returned.

A promising prompt can be run against a dataset of real customer questions to check that it holds up beyond a single example. Once a model and prompt are chosen, the "Get Code" option copies the configuration into the application. The same Fireworks AI models can then power the evaluators that monitor it in production.

Availability and what it means

According to Datadog, the Fireworks AI integration is available today for all Datadog Agent Observability customers. Setup instructions are in the Fireworks AI integration documentation. New Datadog users can start with a 14-day free trial, and those new to Fireworks AI need to sign up for an account to get an API key.

For teams already running open models on Fireworks AI, the change means evaluation and prompt testing can sit in the same place as their traces and token usage, using models from an account they already control.