LangChain: OpenWiki memory made an agent 5x more accurate on cross-source questions

Why LangChain built a compiled wiki

Agents that work across policies, manuals, wikis, tickets and code have to navigate knowledge scattered over systems and teams. LangChain argues that finding a relevant passage is only the start: agents must also decide which rules apply, what has changed and how one source affects another. Without a persistent memory layer, the company says, agents often rebuild that understanding on every question, which is expensive and error-prone.

The problem is especially sharp in banking and insurance, where policies and guidelines frequently amend or supersede one another. Standard RAG pipelines typically use semantic or hybrid search to retrieve isolated chunks and leave the agent to work out how they relate.

OpenWiki is an open-source CLI built on Deep Agents. It synthesizes source material into linked pages that explain key concepts and their relationships, with citations back to the sources. LangChain says it first built the tool so coding agents would not have to rediscover architecture and code behavior on every task.

Each OpenWiki statement rests on a claim, an assertion linked to supporting source lines and their hash. When those lines change, the hash changes, so OpenWiki can flag affected claims without a model call. Only those claims are sent to a model for confirmation, revision or retraction.

The test: 111 insurance documents

To evaluate the approach, LangChain built a synthetic insurance library of 111 long-form documents. It chose insurance because policies and endorsements often reference, amend and override one another. The team created a ledger of 972 facts, controlled where each should appear and how documents referenced each other, and generated the surrounding prose with an LLM. It then mixed in outdated versions, contradictions, indirect reference chains, varied wording and very long documents.

The same retrieval agent was tested in two conditions: Search, using keyword search over the original documents, and Wiki, with access to an OpenWiki-compiled wiki. Model, instructions, documents and questions were held constant. There were 80 simple lookup questions and 50 connected questions that require following dependencies across documents, each asked three times. LangChain scored accuracy, consistency (overlap of cited evidence across runs) and cost and latency.

OpenWiki compiled the library into a wiki roughly one-tenth its original size for $4.15. Adding 17 documents later cost about $0.09 each. The reported results exclude compilation and update costs.

Where the wiki helped and where it did not

On simple lookups, search and wiki performed similarly: 0.84 versus 0.83 accuracy, 0.72 versus 0.74 citation overlap, and $0.020 versus $0.021 per correct answer. Latency rose from 9 to 12 seconds with the wiki.

On connected questions, accuracy rose from 0.10 with search alone to 0.52 with the wiki. Cost per correct answer fell from about $0.55 to $0.19, as the compiled context let the agent find relationships with fewer tool calls.

Citation consistency remained a weakness. Overlap dropped from 0.74 on simple lookups to 0.25 on connected questions. LangChain notes that this metric does not measure correctness, but changing evidence can change an answer, especially when sources conflict, and teams need answers they can reproduce and audit.

Adding an evidence graph

To make retrieval more repeatable, LangChain added a lightweight evidence graph that turns wiki citations into explicit, typed edges between claims, wiki pages, paragraphs and topics. It was built in two steps. A deterministic process splits each document into paragraphs with unique identifiers, so claims link to exact passages rather than whole documents. Paragraphs, claims and pages are then connected with typed edges: claims cite paragraphs, newer documents supersede older ones, and sections refer to other documents.

Because wiki claims already carry text citations, edges such as refers_to and supersedes can be created without another model call. According to LangChain, the graph took under a second to build and resolved every explicit cross-reference in the test library. The company says edges derived from the wiki still depend on the accuracy of its model-generated citations.

Two tools let the agent navigate the graph: locate finds a starting point matching the question, and expand follows its connections to supporting passages. Compared with the wiki alone, the graph doubled citation consistency, improved accuracy by 27% and cut cost per correct answer by 58%, LangChain reports.

What it means for teams building agents

LangChain's conclusion is to save compiled memory for repeated cross-source reasoning. For simple lookups, the agent without memory was just as accurate and faster. Where questions depend on relationships between documents, the company says the wiki captures domain knowledge and the evidence graph adds more accurate and more repeatable retrieval. The results come from a synthetic insurance corpus built by LangChain, so they describe its own benchmark rather than real enterprise libraries.