AI review finds major gaps in how medical studies share code

An MIT and Dana-Farber Cancer Institute-led team used large language models to examine nearly 4,000 clinical prediction studies and hundreds of code repositories. The review found that code sharing remains uncommon, and that public code often lacks the documentation needed for scrutiny and reuse.
- The team reviewed nearly 4,000 clinical prediction studies.
- Only about 1 in 8 studies reported sharing code.
- Public repositories varied widely in documentation and reuse information.
- Human reviewers checked 500 papers and a separate repository sample.
- The findings are informing the TRIPOD-Code reporting guideline.
A large-scale review of code sharing
The researchers examined studies that developed, updated, or validated clinical prediction models, which estimate patient outcomes such as the risk of developing a disease. Because these models can inform patient care, access to the underlying methods and code can help researchers understand how patients were selected, how missing values were handled, and how results were evaluated.
The team identified almost 4,000 relevant studies citing the TRIPOD or TRIPOD+AI reporting guidelines. The papers came from PubMed Central’s open-access collection. An LLM helped screen the papers, identify statements about code availability, and extract links to repositories. Software then retrieved the linked repositories for assessment.
The review examined more than whether a paper included a link. It assessed features including documentation, software requirements, and terms of reuse. Human reviewers independently checked 500 papers and a separate sample of repositories, resolving disagreements before comparing their assessments with the automated results.
Only about one in eight studies shared code
Only about 1 in 8 studies reported sharing code, according to the review. The contents of the repositories varied widely, from collections that were “extremely well-organized and -documented” to those that offered “almost no guidance at all,” Thomas Sounack, the study’s first author, said in the MIT News report.
Among journals, the proportion of studies sharing code ranged from zero to more than 70 percent. Some journals with low sharing rates had written code-sharing policies, suggesting that policies alone may not be enough to motivate researchers to publish their code.
The researchers also found that classification errors remain possible, particularly when judging the adequacy of documentation. Agreement between human reviewers and the automated results was stronger for concrete features, such as whether a license file was present, than for assessments of documentation quality. The authors therefore emphasize broad patterns rather than treating every count as exact.
TRIPOD-Code aims to set clearer expectations
The findings are informing TRIPOD-Code, a reporting guideline under development that is intended to clarify what researchers should share. It builds on TRIPOD and TRIPOD+AI, checklists covering how clinical prediction models are developed and evaluated.
Tom Pollard, a research scientist at MIT and the study’s senior author, said the team wants to provide “practical guidance on how to share code” and give journals “clearer expectations to communicate to authors.” The guidance is being developed through an expert review process focused on which elements of shared analytical code should be considered essential.
The study also identified examples of effective practice. Some researchers provided synthetic datasets when original patient records could not be shared publicly, giving others a way to work with the code while protecting access to the underlying patient data. For users of medical research, the proposed guidance could make it easier to assess published models and determine whether their code can be reused.

