AWS introduces UAEF for cross-framework AI agent evaluation

One evaluation layer across agent frameworks

Teams building agents on AWS may use frameworks such as LangGraph, LangChain, Strands Agents, and Amazon Bedrock AgentCore. Because each records messages, tool calls, and handoffs differently, evaluation systems built around one framework's trace format can require substantial rebuilding when teams change frameworks or use several of them.

UAEF addresses that problem with adapters. An adapter reads an agent's native execution record and converts it into a canonical format called the AgentTrace. UAEF then applies the same scoring and reporting process regardless of which supported framework produced the trace.

The system can work with existing traces from past runs, logs, or observability stores. It can also invoke an agent through a runtime Amazon Resource Name or an HTTP endpoint, run it against dataset inputs, collect the response, and adapt the result. AWS says UAEF can run locally, behind an HTTP endpoint, or on a managed runtime such as Amazon Bedrock AgentCore, with traces in Langfuse or without captured traces.

36 metrics across seven dimensions

UAEF evaluates agents across seven quality dimensions: tool calling, response quality, responsible AI, performance, multi-turn conversation, multi-agent coordination, and reasoning. These areas cover whether an agent selects and uses tools correctly, produces relevant and grounded answers, handles safety concerns, manages operational performance, retains context, coordinates work, and maintains consistent logic.

Each dimension uses one or more of 36 built-in metrics. Every metric returns a normalized score from [0, 1], with 1 representing the best result. Datasets can include ground-truth answers, although this is optional. To evaluate tool calling, the ground truth must also specify the expected tools, their order, and their arguments.

The metrics include deterministic measures that reproduce exactly on repeated runs, as well as large language model judge metrics for qualities such as relevance and safety. AWS says teams can add custom metrics or use integrations with RAGAS and DeepEval. Judge content is treated as untrusted input, and a score is rejected if its required evidence cannot be found verbatim in the evaluated content.

Stored experiments support version comparisons

When a team persists an evaluation, UAEF records it as an experiment in Amazon DynamoDB and Amazon Simple Storage Service, also known as Amazon S3. DynamoDB stores experiment metadata for listing and comparison, including the evaluation count, average score per metric, and overall average. S3 stores the full results as a JSON file, including per-metric scores and the judge's reasoning.

An experiment can contain multiple runs. Each run can record the framework, model, tools, system prompt, parameters, a configuration snapshot, and a Git commit. Teams can designate a baseline and compare later versions against it, helping identify regressions or improvements linked to changes in a prompt, model, or tools. AWS says customers across 12 organizations in the telecom, healthcare, and financial services industries used UAEF to evaluate agents and compare versions before release.

For users building agents on AWS, the system's significance is its shared evaluation layer. Instead of tying metrics and historical results to one framework, teams can use a common trace, consistent scores, and stored experiments to guide the next development cycle.