Artificial intelligence news.
Straight from the original announcements.

Kensho explains why static benchmarks fail for financial AI retrieval

October 6, 2026, 15:00By the AI News Bites Editorial TeamSource: Kensho (S&P Global)
Kensho explains why static benchmarks fail for financial AI retrieval
Image: Kensho (S&P Global)

Kensho, the AI team at S&P Global, says fixed benchmarks cannot reliably show whether a retrieval agent found the right evidence in financial data that never stops changing. In an engineering post, it describes a custom evaluation harness that freezes each question in time to check whether the agent returned trusted data.

Key points
  • Kensho says fixed benchmarks like MS MARCO and BEIR assume a fixed corpus and predefined correct results, which production financial data does not offer.
  • Its evaluation uses a custom harness that freezes each question in time to check whether the agent returned trusted data.
  • It examines five surfaces: chunk content, chunk metadata, chunk identities, agent trace and generated answer.
  • An LLM-judge A/B approach is described as "promising but not yet proven" and is not relied on in practice.
  • Kensho says maximum stability is not the goal, since forcing consistency would limit the agent's reasoning.

A retrieval problem that keeps moving

Financial professionals work with structured datasets such as company financials and market data, and with unstructured sources like earnings call transcripts, investor presentations and corporate filings. According to Kensho, the challenge as more analysis shifts to AI applications and agentic workflows is surfacing precisely the right information, grounded in trusted and verifiable data.

The S&P Global AI Data Portal offers two retrieval methods built to work with any AI and agent architecture. For complex, open-ended research, the path is Adaptive Retrieval. It breaks a natural language question into sub-queries and routes each to a specialized agent built with subject-matter expertise for a particular dataset. The agents return citation-backed responses. That design raises the question the post sets out to answer: how do you know whether an agent retrieved the best available evidence and produced a trustworthy response?

Five surfaces to evaluate

Kensho's unstructured search agents return verbatim chunks from grounded documents rather than finished answers. The team looks at several surfaces, each giving a different view of quality:

  • Chunk content: the retrieved evidence is compared with gold snippets, text spans that a subject-matter expert labels as the evidence a question should return. Without special handling, these go stale as new documents are published.
  • Chunk metadata: expected companies, dates, fiscal periods and document types tend to stay stable as the corpus changes. They show whether retrieval stayed in scope, though not whether the content answers the question.
  • Chunk identities: comparing chunks across repeated runs reveals stability. Overlap measures consistency, not correctness, since different chunks can be equally valid.
  • Agent trace: queries, tool calls and search decisions show whether the agent followed its instructions. Because of non-determinism, a single gold trace is too restrictive, but developers can check that a trace follows an expected general shape.
  • Generated answer: this goes beyond the agent's output contract in Adaptive Retrieval, but serves as a proxy for the evidence returned and can make comparisons easier.

Why MS MARCO and BEIR are not enough

Structured data has its own answer key: a revenue figure is right or wrong. Unstructured financial information has none, Kensho argues. Evidence is scattered across prose, from management commentary on a call to disclosures in regulatory filings, and finding it can require interpreting time periods, companies, document types and financial terminology.

Traditional information-retrieval evaluation starts with a fixed document collection, a set of questions and relevance judgments, then scores systems with metrics such as recall@k, mean reciprocal rank and nDCG. Benchmarks like MS MARCO and BEIR made this framework invaluable for research, according to the post, but it assumes the corpus is fixed and the correct result can be defined in advance. Production financial data meets neither condition. New documents arrive continuously, so gold labels decay, and many questions allow more than one reasonable interpretation, so even experts may disagree on the ideal answer.

Kensho still sees public benchmarks as a useful starting point for comparing methods in the abstract. Its warning is that a system can score well on a static reference set while performing poorly in production.

Comparing systems without gold labels

One option the team explored is a form of unsupervised A/B testing. An LLM writes an answer grounded on the chunks retrieved by each of two system versions, and a judge picks the answer that better addresses the question. Answers stand in for the retrieved content because comparing two lists of chunks is ill-defined: they differ in count, length, granularity, overlap and order, and metrics like nDCG assume a human scanning top to bottom, not a downstream LLM.

The approach has its own weak point, the judge. According to Kensho, LLM judges can be swayed by verbosity, formatting, writing style and even the order in which answers appear. Order reversal and length analysis help expose these biases, but a judge is only trustworthy to the extent it agrees with subject-matter experts, which requires labeled comparisons and expert time to tune its prompt. Pairwise runs are also slower because they need multiple LLM calls. The company says it treats the method as "promising but not yet proven" and does not rely on it in practice.

The post also addresses consistency. Agentic retrieval is a chain of non-deterministic LLM-driven decisions, so the same question asked twice can produce different behavior and results. Kensho writes: "Maximum stability is not the goal." For open-ended questions, it says, forcing consistency would constrain the agent's reasoning and lower answer quality to buy lower variance, so the aim is a balance. The source text breaks off before the team finishes describing that balance.

What it means for teams building on financial data

The post is a look at how one large data provider tests its own retrieval agents, and it is candid about limits. Kensho does not claim a single metric settles quality. It describes evaluation built around comparison, variability, evaluator bias and time. For teams building research agents on top of fast-changing document sets, the practical message is that stable signals such as metadata scope checks and trace shape can complement gold snippets, which decay as new filings and transcripts appear.

Related news

SAS and Posit Partner on Governed AI for Life Sciences
BusinessNovember 16, 2026, 09:00

SAS and Posit Partner on Governed AI for Life Sciences

SAS and Posit PBC have formed a partnership to give life sciences organizations more flexibility in using SAS and open-source technologies, initially for clinical…

Canada’s LNG Ambition Could Unlock C$1 Trillion in Investment, Study Says
ResearchNovember 12, 2026, 12:53

Canada’s LNG Ambition Could Unlock C$1 Trillion in Investment, Study Says

Canada’s ambition to export 100 MMtpa of liquefied natural gas by 2050 could support nearly C$1 trillion in investment and almost a quarter million jobs each year…

Four GPU-heavy AI workloads reveal different infrastructure needs
Chips and infrastructureNovember 4, 2026, 17:30

Four GPU-heavy AI workloads reveal different infrastructure needs

Four AI workloads, from factory-floor computer vision to quantum computing simulation, are placing distinct demands on GPU infrastructure, according to research…