Firecrawl Publishes Legal Search Benchmark With 581 Research Requests

A benchmark built from customer searches

The benchmark covers requests for statutes, regulations, court opinions, agency guidance, tax rulings, bills and local ordinances. Each request asks for one official document, and Firecrawl says the questions are grounded in what its customers actually search for.

The release, called benchmark-legal-retrieval, contains 290 public requests and 291 held-out requests. The public portion includes its answer key and scoring code on GitHub. Each request has one official answer, along with official aliases and known near matches such as the same text on a mirror site or an official page closely connected to the answer.

Firecrawl says the benchmark is built around four requirements: real demand, an official document as the answer, deterministic grading and a way to keep the dataset fresh. Instead of using a model to grade results, it scores each engine by exact matches against a frozen canonical URL. That means a repeat run gets the same score, according to Firecrawl.

Firecrawl reports a lead on first-result accuracy

In Firecrawl's 3 Nov 2026 run, the Government & Legal Index placed the official page first on 76.2% of requests. Firecrawl reports 59.6% for Exa, 54.2% for Parallel and 71.4% for Claude Code using its own web search.

The index found the official page somewhere in the top 10 on 96.6% of requests. That compares with 90.5% for Parallel and 89.5% for Exa in the same run. Firecrawl also says a contradicting page appeared in the index's top 10 on 9.6% of requests, statistically level with Exa at 9.6% and Parallel at 9.3%.

Firecrawl reports that the index's searches are free, require no credits and do not require an API key. The company lists Exa at $0.007 per request and Parallel at $0.005. A complete pass over all 581 requests would cost $4.07 on Exa, $2.91 on Parallel and nothing on the index.

A weekly evaluation and improvement loop

Firecrawl describes the benchmark as one half of a weekly data flywheel. Each version begins with a sample of recent searches on the Government & Legal Index, with at most one request per customer team and no requests from organizations with zero data retention. A model rewrites each search into a twin that keeps the same jurisdiction, document type, task and phrasing style while asking for a different official document.

The company then builds an answer key from official sources, freezes the questions and key, and runs every engine on the same set. For each miss, Firecrawl checks whether the correct page is absent from the index, ranked too low or blocked by a link that does not load. The search team uses that information to make fixes, while the next benchmark version is scored before its misses reach the team.

The loop has already turned eight times inside Firecrawl, but this is the first version the company has published. Firecrawl says the loop will continue weekly, with a new public benchmark version shipping with each new release of the Government & Legal Index.