AI Drafts Doctor Reply Letters From Hospital Records, but Experts Still Rate Human Writing Higher

Medical documentation is one of the quiet drains on clinical time, and reply letters sent from specialty departments back to primary care are a recurring part of that workload. A new study in the Journal of Medical Systems reports the development and evaluation of an instruction-tuned large language model designed to generate these letters automatically by drawing on five types of electronic medical record documents: admission notes, progress notes, procedure notes, operative notes, and discharge summaries.

The researchers, led by Eunbeen Jo, Yebin Lee, and Eunbi Cho with senior author Sanghoun Song, trained the system on 8,527 hospitalization episodes from the cardiology and thoracic surgery departments of a tertiary hospital. A separate set of 105 episodes was held out for testing. The base model was Mistral-Nemo-Instruct-2407, fine-tuned on an instruction-tuning dataset generated with GPT-4.1, meaning a larger commercial model was used to create the training examples that shaped the smaller open-weight model’s letter-writing behavior.

Evaluation used a dual framework combining expert assessment with a GPT-based automated evaluation, and the two judges disagreed in an instructive direction. Human experts rated human-written letters significantly higher in accuracy (4.77 vs. 4.21) and appropriateness (4.90 vs. 4.63), with both differences statistically significant at p < 0.001. The GPT-based evaluation, by contrast, scored the AI-generated letters higher across all dimensions, a divergence that highlights how automated judges and clinical experts can weight the same texts differently.

To reconcile this, the team tested a hybrid scoring approach integrating expert and GPT-based evaluations, which improved score consistency among evaluators. In an exploratory comparison using a common weighting, the largest proportion of cases was classified as comparable between the human-written and AI-generated letters across all quality dimensions, suggesting the machine drafts were often close to acceptable even where humans retained an edge on accuracy and appropriateness.

Experts also rated workflow suitability, with a mean of 4.30, indicating positive expectations for practical implementation. The authors frame the model as a supportive tool for drafting clinical correspondence rather than a replacement for clinician review, and they caution that its safety and its actual impact on documentation burden require prospective evaluation in real-world clinical workflows before any deployment conclusions can be drawn.

Subject of Research: Development and expert-versus-GPT evaluation of an instruction-tuned large language model that generates specialty-to-primary-care reply letters from five electronic medical record document types.

Article Title: Development and Evaluation of an Instruction-Tuned Large Language Model for Reply Letter Generation from Multiple Clinical Document Types

Article References: Jo, E., Lee, Y., Cho, E., Kim, S., Park, E., Ryu, H., Joo, H. J., & Song, S. (2026). Development and Evaluation of an Instruction-Tuned Large Language Model for Reply Letter Generation from Multiple Clinical Document Types. Journal of Medical Systems, 50(1), Article 157. https://doi.org/10.1007/s10916-026-02483-1

Image Credits: AI Generated

DOI: 10.1007/s10916-026-02483-1

Keywords: large language models, instruction tuning, clinical documentation, reply letters, electronic medical records, Mistral-Nemo-Instruct-2407, GPT-4.1, cardiology, thoracic surgery, expert evaluation, tertiary hospital, Development