AI Checklist Helps Radiologists Read Tricky Heel Surgery X-Rays Faster and More Accurately

When surgeons repair a shattered heel bone, the follow-up X-rays are notoriously hard to read. The calcaneus, the bone that forms the heel, is a labyrinth of overlapping plates, screws and healed fracture lines, and even experienced radiologists can disagree about whether a postoperative film shows a complication that demands clinical escalation. A new multicenter study published in BMC Medical Imaging suggests that a multimodal large language model, the same family of artificial intelligence systems that power modern chatbots, can meaningfully sharpen how doctors interpret these challenging images, while also exposing some sobering caveats about how AI assistance should be tested.

The research, led by Shaobin Wang and Hai Li of the University of Science and Technology of China and Hefei First People’s Hospital, together with colleagues across three hospital campuses in Hefei, China, set out to answer a deceptively simple question: does a pre-generated AI draft report and structured checklist actually help radiologists classify postoperative calcaneal radiographs more accurately than reading them unaided? The team recruited six radiologists and asked them to interpret 73 cases of postoperative calcaneal fixation, generating 438 case-reader observations in each of two reading sessions. Postoperative computed tomography interpreted by expert consensus served as the pragmatic reference standard against which every X-ray judgment was measured.

The study’s design was deliberately careful about the mechanics of AI assistance. In the first session, radiologists read each case entirely on their own, assigning a risk category and recording their confidence and reading time. Then, after a washout period of more than four weeks designed to blur memory of the first pass, the same readers returned for a second session in which they reviewed and finalized a fixed, pre-generated checklist and draft report produced by GPT-5, a multimodal large language model capable of processing both images and text. The readers worked within the same software interface in both sessions, and the order of cases was rerandomized between sessions to reduce recall effects.

The headline result is striking. Pooling medium- and high-risk cases into a single binary category of cases requiring escalation, reader accuracy rose from 71.5 percent in the unaided session to 80.1 percent in the AI-assisted session, a difference of 8.7 percentage points with a cluster-bootstrap 95 percent confidence interval of 1.1 to 17.4. Expressed as an odds ratio, the improvement was 1.97 with a confidence interval of 1.37 to 2.84, and the overall effect reached statistical significance at P less than 0.001. In practical terms, the AI-assisted workflow roughly doubled the odds that a reader’s escalation decision matched the CT-based expert consensus.

The gains were not evenly distributed across the diagnostic spectrum, however. Sensitivity, the ability to correctly flag cases that truly required escalation, jumped dramatically from 54.2 percent to 79.2 percent, meaning the AI checklist helped readers catch far more of the genuinely concerning cases. Specificity, by contrast, slipped from 85.0 percent to 80.9 percent, indicating a modest rise in false alarms among cases that were actually fine. At the stricter exact-high-risk threshold, the picture was less encouraging: sensitivity was 45.0 percent in the unaided session and 43.3 percent in the assisted session, showing that the AI did not help readers identify the very highest-risk subset with any greater precision.

Speed and calibration also shifted in the AI-assisted session. Mean recorded reading time fell from 378.6 seconds per case to just 98.4 seconds, an adjusted ratio of 0.257, meaning readers finished the assisted pass in roughly a quarter of the time. The mean Brier score, a measure of how well readers’ stated confidence matches actual correctness where lower is better, improved from 0.246 to 0.177, a difference of 0.069 with a confidence interval of 0.007 to 0.147. Together, these numbers paint a picture of readers who were faster, better calibrated and more likely to catch true problems when working from the model’s structured draft.

The authors are unusually candid about the study’s central limitation, and it is one that anyone following medical AI should understand. Because the AI-assisted session was always conducted second, the design cannot separate the genuine effect of the model-generated checklist and draft report from the effects of reading order, practice, or simple familiarity with the case mix and interface. A reader who performs better on a second pass might be benefiting from experience rather than from the AI itself. The washout period and rerandomized case order were meant to blunt this problem, but the session order was not counterbalanced, meaning some readers could not have gone first with AI assistance. The authors explicitly call for prospective randomized or counterbalanced validation across independent institutions before the approach is adopted clinically.

Why does this particular imaging task matter so much? Calcaneal fractures are among the most disabling of all orthopedic injuries, often resulting from falls from height or motor vehicle collisions, and surgical fixation with plates and screws is common. Postoperative radiographs must confirm that hardware is intact, that the bone is healing in acceptable alignment, and that no complications such as hardware failure, loss of reduction or joint penetration have emerged. The subtlety of these findings, overlaid as they are with metalwork and complex bony anatomy, makes them a demanding test case for AI assistance, far harder than the chest X-ray or bone-age tasks that have dominated earlier radiology AI studies.

The study also stands out for its methodological transparency. The authors report their work against the STARD and STARD-AI standards for diagnostic accuracy studies, completed the CLAIM checklist for artificial intelligence in medical imaging, and released their analysis code as a portable R script alongside supplementary tables, reliability plots and reader-trajectory figures. Statistical analysis relied on generalized linear mixed-effects models to account for the clustered structure of the data, in which multiple readers judged the same cases and each reader judged multiple cases. The work was funded by the National Natural Science Foundation of China and the Anhui Provincial Key Research and Development Project, with the funders having no role in study design or interpretation, and the authors declare no competing interests.

The broader lesson for medicine is twofold. On one hand, the study offers some of the most concrete evidence yet that multimodal large language models can function as genuine cognitive partners in musculoskeletal radiology, not just as novelty demos, boosting sensitivity and slashing reading times on a genuinely difficult task. On the other hand, it is a masterclass in why AI evaluation studies must be designed with the same rigor as drug trials. A promising 8.7 percentage point accuracy gain, a fourfold speed increase and better-calibrated confidence are exactly the kind of results that could transform reporting workflows, but the authors’ own warning stands: until counterbalanced, prospective, multi-institutional trials confirm the effect, the AI checklist should be seen as a compelling candidate, not a proven prescription.

Subject of Research: Multimodal large language model assistance for interpreting postoperative calcaneal fracture radiographs in a multicenter multi-reader study

Article Title: Multimodal large language model assistance in interpreting postoperative calcaneal radiographs: a multicenter multi-reader study

Article References: Multimodal large language model assistance in interpreting postoperative calcaneal radiographs: a multicenter multi-reader study. (n.d.). https://doi.org/10.1186/s12880-026-02865-4

Image Credits: AI Generated

DOI: 10.1186/s12880-026-02865-4

Keywords: artificial intelligence, large language models, radiology, calcaneal fracture, postoperative imaging, structured reporting, human-AI collaboration, diagnostic accuracy, GPT-5, medical imaging, Brier score, multicenter study