New Relic Adds AI Evaluation to Score LLM Quality Inside Traces

What AI Evaluation does

New Relic says enterprise teams moving generative AI from pilot to production face a visibility gap. Traditional monitoring measures throughput and latency, but it misses semantic failures, leaving teams blind to hallucinations, PII leaks and unexpected model drift, according to the company.

AI Evaluation is built as a platform capability within New Relic AI Observability. It is powered by an asynchronous LLM-as-a-judge service that evaluates sampled prompt payloads and model completions against predefined quality and security parameters.

Each evaluated interaction receives three outputs:

• A normalized score from 0 to 1
• A pass/fail status based on customizable thresholds
• An explanatory reasoning string

These results are returned to the platform and attached as attributes to the original distributed traces. New Relic says this lets engineers move from a low-quality response score to the specific prompt span, vector database retrieval call or backend service behind the problem, in a single view.

Built-in checks for security and RAG quality

New Relic says AI Evaluation includes out-of-the-box evaluators that need no setup or configuration. The security and compliance checks cover three areas:

• Prompt injection and jailbreak defense: identifies adversarial user instructions and attempts to bypass safety guardrails.
• Data leakage mitigation: scans incoming payloads and completions for sensitive data, flagging personally identifiable information (PII).
• Toxicity and bias screening: detects harmful, offensive or abusive language, as well as unfair treatment based on protected attributes.

For agents and Retrieval-Augmented Generation (RAG) systems, the service evaluates retrieval and generation separately to help identify which component failed. Faithfulness, described as hallucination tracking, measures whether responses are factually consistent with the retrieval context. Answer and contextual relevancy measure how well retrieved information matches the user query and whether the final response addresses the prompt.

Linking quality to cost, with configurable sampling

The company also positions the tool as a way to manage model spending. By connecting response scores to infrastructure consumption and token costs, teams can check whether expensive, high-parameter models deliver better accuracy and relevance than faster, lower-cost alternatives. New Relic says organizations can use this to swap out underperforming LLMs and lower token spend without hurting answer quality.

To limit the overhead of evaluation itself, platform operators can set granular sampling rules on live traffic. New Relic gives the example of running toxicity or data leakage checks on 10 percent of transactions rather than every interaction. The asynchronous processing is meant to keep operating costs controlled while maintaining continuous safety oversight.

Who it is for

New Relic names three stakeholder groups. AI/ML and LLMOps engineers get evaluation scores tied to traces, replacing log spreadsheets and manual reviews, and can check whether changes to system prompts or RAG retrieval logic improve relevance or cause regressions such as hallucinations. Platform engineers, SREs and DevOps leads can correlate quality metrics with token costs and set sampling rules to protect platform performance.

The company contrasts the product with standalone LLM observability tools that store evaluation scores in disconnected silos. Its argument is that evaluating the whole transaction, including upstream prompts, vector database retrievals and downstream compute, ties probabilistic quality scores to software root causes. New Relic also claims that automating guardrails removes manual log reviews that slow deployments, allowing teams to ship generative AI applications faster. Pricing and availability details were not included in the source text.