UChicago Team Uses Bayesian Models to Check LLM Readings of Babylonian Diaries

A grant to extract and verify ancient observations

Aaron Schein, an assistant professor of statistics and data science, and Chris Wolfram, a PhD student in computer science, both at the University of Chicago, are developing a method to extract and check astronomical observations from the Babylonian Astronomical Diaries. Schein received the grant from HAVI. The pair are working with John Steele, a professor at Brown University and a leading expert on Babylonian astronomy.

The Diaries were written in Akkadian on cuneiform tablets over roughly six centuries. The Babylonian astronomers tracked the relative positions of the Moon, planets and stars, and also noted weather, commodity prices, river levels and major events. According to the university, the records are the product of the longest-running data collection project in human history.

The project is part of a HAVI initiative that awards $10 million to 19 research teams using AI in the humanities, including the study of history, art, language and music. The latest awards bring HAVI's total funding commitment to more than $21 million over two years.

Catching LLM mistakes with a verification loop

LLMs can help researchers work through large collections of historical texts, pulling out information that would otherwise take years to compile. Their answers are not always reliable, though. "LLMs are very good at extracting these kinds of structured observations from text, but they make mistakes, and they hallucinate," Schein said. "Our question was how could we use them here without simply trusting them?"

The team's answer is to use modern astronomy as an independent check. Their method combines LLM-extracted observations with a Bayesian statistical model that compares each observation against where the Moon, planets and stars actually were that night, according to modern astronomical calculations.

When an observation does not match the model's prediction, the discrepancy becomes a clue. The problem might lie in:

• the AI's extraction
• a translation
• the digitization process
• the modern chronology used to convert Babylonian dates to our calendar
• a mistake by the ancient observer

The team uses what the model flags to fix errors and refine the extraction pipeline. The cleaner data supports a richer model, and the loop is repeated to remove as many errors as possible. "We can use the power of LLMs, but within this loop that allows us to verify the observations and build credibility for them," Schein said.

Early results

According to the university, the model has already produced a precise estimate of the length of the cubit as the Diarists used it. The cubit is an angular unit whose exact value has long been debated. The model also produced the first quantitative estimates of what the Diarists meant by phrases such as "the last part of the night."

It also independently rediscovered known errors in an older chronology used to convert Babylonian dates to the modern calendar, which suggests it can catch mistakes that scholars have had to find by hand. The same checks caught an issue in an astronomical software library the team used for the comparison. The model additionally fills in data lost to damaged tablets.

What it could mean for historians and beyond

The immediate goal is a comprehensive database of astronomical observations from the Diaries, together with open-source code that other researchers can build on. The model could also help date tablets whose dates are missing or uncertain, and flag tablets whose assigned dates may be wrong.

Because the records also hold information on commodity prices, weather and other events, a more complete, searchable dataset could eventually help researchers study long-term changes in prices and climate, among other questions.

Schein sees the method as possibly just as important as any single historical finding. The same approach, using domain knowledge and statistical models to check information extracted by AI, could in principle be applied to any collection of records describing phenomena that can be independently modeled. "There are only a couple of such windows into the past," Schein said. "So it makes sense to clean them as best we can."