FantasyBench: Firecrawl Alexandria Tops AI Agent Fantasy Data Test at 80.5%
Firecrawl says its Alexandria data library scored 80.5% on FantasyBench, a new public benchmark of how accurately AI agents answer fantasy football questions. Native web search reached 71.7% and the league-aware Flaim (STACKED MCP) server 71.3%, according to Firecrawl's own write-up.
- Firecrawl's FantasyBench uses 272 real questions on the 2026 NFL season: 150 past-game and 122 Week 6 projection questions.
- Firecrawl Alexandria scored 80.5%, ahead of native web search at 71.7% and Flaim (STACKED MCP) at 71.3%.
- On projections, Alexandria scored 77.9% against 60.7% for web search and 61.5% for Flaim.
- Alexandria's returned values were correct 98.8% of the time, versus 85.0% for web search.
- FantasyPros Coach AI and SportsDataIO were not tested.
What FantasyBench tests
Firecrawl describes FantasyBench as the first public benchmark that measures how accurately AI agents handle real fantasy questions. It contains 272 questions drawn from agent searches on the 2026 NFL season. Of these, 150 cover past-game stats, usage and injury designations from Weeks 1 to 5. The other 122 cover live Week 6 projections and rankings.
Past-game answers are graded against nflverse, cross-checked against Sleeper (416 of 417 values agree) and ESPN (79 of 79 scores). Projection answers are graded against pages saved from seven sites: CBS, Draft Sharks, StartWho, FantasyPros, Fantasy Football Calculator, FantasyData and FFToday. A model copies values out of each answer but does not judge them. A frozen program compares them with the answer key, so identical answers always receive the same score.
All three tested tool types, data providers, web search and league-aware MCP servers, ran on the same agent (claude-opus-5-5), which makes the scores directly comparable, according to Firecrawl.
How the tools scored
Firecrawl Alexandria answered 219 of 272 questions fully correctly, for 80.5%. That is 8.8 percentage points ahead of native web search, a gap Firecrawl calls statistically significant (p = 0.005, exact McNemar test). Flaim scored 71.3% and is described as statistically tied with web search.
The difference comes from projections. On the 122 Week 6 projection questions, Alexandria scored 77.9%, web search 60.7% and Flaim 61.5%. On past-game stats the tools clustered around 80%, with Alexandria at 82.7% and web search at 80.7%.
Value accuracy also diverged. When Alexandria returned a number, it was correct 98.8% of the time. Web search was right 85.0% of the time, so 15% of the numbers it reported were wrong. Median answer time was 46.8 seconds for Alexandria and 65.0 seconds for web search.
Firecrawl attributes the gap to how data reaches the agent. Web search reads a page and asks the model to pull numbers from tables, while Alexandria returns structured records with the requested fields. Alexandria's fantasy coverage includes five projection sources (CBS, Draft Sharks, StartWho, Fantasy Football Calculator and FFToday), plus injury and news reports, rest-of-season boards and league integration with ESPN, Sleeper or Yahoo.
What it means for fantasy players
The post is published by Firecrawl, which also makes the top-ranked tool, so the ranking reflects the company's own benchmark. Firecrawl lists FantasyPros Coach AI and SportsDataIO as not tested, and includes them only because they come up in community discussions.
For users, the numbers suggest a split. Firecrawl says native web search is fine for quick questions, injury checks and general rankings, but advises against it for setting a lineup or making trade decisions, where a wrong number can cost a matchup. Alexandria is positioned for accuracy-critical work, with trade-offs the company states itself: it requires a Firecrawl account and credits, a keyless free mode covers evaluation only, and some niche projection sources are not yet included.
