AI Matches Experts Moderately Well in Scoring Simulation Debriefings

A personalized ChatGPT model scored recorded simulation debriefings much the way human experts do, but not close enough to replace them, according to a pilot study published in Advances in Simulation. The research, led by Andrés Ferre and colleagues including Soledad Armijo-Rivera, compared how well a tuned large language model and human raters agreed when grading debriefing quality with the Debriefing Assessment for Simulation in Healthcare, or DASH, a validated instrument used in healthcare simulation education.

The study addressed a practical bottleneck. Debriefing is central to healthcare simulation, yet assessing its quality normally depends on expert raters and consumes substantial faculty time. Large language models could, in principle, support transcript-based quality review, but evidence has been lacking on how well their scores agree with experts and on the limits of judging a debriefing from a transcript alone. The team therefore ran a prospective, multicentric pilot study with a mixed-methods design, analyzing 33 high-fidelity simulation debriefings.

Each transcript was scored on DASH items 2 through 6, which together range from 5 to 35 points. Human experts produced 150 evaluations, while personalized GPT-4.0 and GPT-4.5 models produced 345 evaluations in total. The researchers used an adaptive design to choose the better-performing AI model, requiring internal consistency of at least 0.7 on Cronbach’s alpha and inter-evaluator concordance of at least 0.5 on the intraclass correlation coefficient. GPT-4.5 clearly outperformed GPT-4.0, achieving an alpha of 0.870 against 0.427, and was selected for the main analysis.

On continuous scoring, the personalized GPT-4.5 model showed moderate concordance with the human experts, with intraclass correlation coefficients ranging from 0.290 to 0.560 and a median of 0.455. When the same ratings were collapsed into coarse categories, agreement looked far stronger: 95 to 100 percent, with prevalence-adjusted bias-adjusted kappa values between 0.900 and 1.000. The authors caution, however, that this categorical agreement came from a dataset in which ratings were predominantly classified as “Good,” meaning nearly everyone, human and machine alike, was placing most debriefings in the same high band.

The qualitative arm of the mixed-methods study used comparative content analysis and revealed a consistent pattern in the written justifications behind the scores. Both the experts and GPT-4.5 frequently identified similar strengths in the debriefings. The experts, though, more often provided specific, critical, improvement-oriented feedback, while the language model tended toward a predominantly positive, descriptive tone.

The authors conclude that in transcript-only scoring of DASH items 2 through 6, a personalized GPT-4.5 model demonstrated strong internal consistency and moderate concordance with expert ratings on continuous scales, while coarse categorical classifications yielded high agreement in a predominantly high-scoring sample. They frame the finding as support for a human-supervised role for large language models in structuring transcript-based review and potentially triaging debriefings for faculty development workflows, rather than substituting for expert assessment.

The study’s scope and limitations are explicit. It was a pilot with 33 debriefings, and the quality distribution of the sample was skewed toward high ratings, which inflates categorical agreement measures. The authors state that future work should test performance in more heterogeneous quality distributions and with multimodal observation beyond transcripts, since a transcript alone cannot capture tone, body language, or other dimensions of a live debriefing.

As an early-access, open-access research article published on 28 August 2026, the paper is citable with a permanent DOI, though the version is subject to further edits before the final Version of Record. For simulation educators, the takeaway is measured: a tuned language model can reliably and consistently structure transcript-based DASH review and flag cases for human attention, but the expert eye, with its sharper critical feedback, remains the standard against which the machine is still being measured.

Subject of Research: Concordance between human experts and personalized large language models in transcript-based DASH scoring of healthcare simulation debriefings

Article Title: Transcript-based DASH scoring with personalized large language models: a mixed-methods concordance study and measurement considerations

Article References: Ferre, A., Gil, D., Giglio, A., García, J., Vergara-Arias, K., Hasbun, P., & Armijo-Rivera, S. (2026). Transcript-based DASH scoring with personalized large language models: a mixed-methods concordance study and measurement considerations. Advances in Simulation. https://doi.org/10.1186/s41077-026-00477-0

Image Credits: AI Generated

DOI: 10.1186/s41077-026-00477-0

Keywords: healthcare simulation, debriefing, DASH score, large language models, GPT-4.5, transcript analysis, human-AI concordance, faculty development, mixed methods, pilot study, Transcript-based, DASH