Artificial intelligence is increasingly being handed the job of judging medical students, and a new study offers one of the most detailed looks yet at how well that judging actually holds up against human experts. Researchers at a single medical school put a commercially available AI simulation platform through a two-phase calibration process and found a striking pattern: the AI matched expert raters almost perfectly on checklist items about concrete behaviors, but stumbled repeatedly on items requiring contextual judgment—exactly the kind of nuanced interpersonal assessment that simulation education is supposed to cultivate.
The study, published in Advances in Simulation, examined SimConverse, an AI-driven simulation-based experience (SBE) platform that lets learners hold spoken conversations with a simulated patient and then receive immediate feedback scored against a rubric chosen by the instructor. Unlike platforms built on third-party models such as ChatGPT, SimConverse relies on proprietary large language models developed in-house. The research team configured a simulated patient: a woman living with excess weight seeking a new primary care provider to discuss health concerns, including weight gain. The rubric was the Weight Sensitivity Instrument, a 20-item checklist designed to assess weight bias-related clinical behaviors, trimmed to 11 items because the AI encounter was purely conversational and involved no physical examination.
The Weight Sensitivity Instrument was originally built for traditional, in-person simulations with standardized patients, and this project marked its first application to an AI-based format. Its items fall into three domains—communication, psychological safety, and physical assessment—with examples ranging from asking about current health concerns before discussing weight to avoiding labels like “obese” or “fat.” The instrument had previously shown good content validity, with a Scale Content Validity Index of 0.91, and strong interrater reliability among instructors scoring nursing students, with Gwet’s AC1 values between 0.90 and 1.0 for 19 of 20 items.
The researchers’ central question was whether AI could apply such a rubric with the same nuance and consistency as human raters, particularly when items differ in complexity or require interpersonal judgment. Prior studies had reported overall AI-human agreement—one team found a Cohen’s kappa of 0.832 across 106 student chatbot interactions, though several individual items fell below 0.6—but offered little insight into how agreement varies across specific item types or how AI scoring can be iteratively improved. The new study set out to fill that gap by documenting the entire calibration process, warts and all.
Phase one was an exercise in iterative prompt engineering: systematically revising the instructions given to the AI for scoring each rubric item. Four researchers and four volunteers, ranging from people with no clinical experience to expert clinicians, generated 31 test interactions in SimConverse. Transcripts were downloaded and scored by a team member with expertise in the instrument, who was blinded to the AI’s scores and demonstrated good intra-rater reliability of their own, with an intraclass correlation coefficient of 0.803. The team then compared AI and expert assessments using percent agreement and Gwet’s AC1, a chance-corrected agreement coefficient chosen because it is more stable than Cohen’s kappa for checklist-style presence-versus-absence items.
The first round of testing revealed both strengths and weaknesses. Eight of the 11 items showed very good agreement, with AC1 values above 0.81. But three items landed only in the moderate range, and in every case the AI was more generous than the human expert, crediting a higher percentage of transcripts with accomplishing the task. One telling example involved an item asking whether the student acknowledged the complexity of managing one’s weight. The AI gave credit to 80.7 percent of transcripts for any general acknowledgment of behavioral challenges, while the human rater credited only 58.1 percent, reserving the point for statements that addressed the specific factors contributing to that complexity. The fix was to add explicit examples to the rubric prompt, instructing that learners should elaborate on the many influences on weight beyond diet and exercise.
After revising the rubric instructions—and also reprogramming the simulated patient to show more reticence about discussing weight, since the character had brought the topic up immediately in many encounters—four researchers produced 10 additional test interactions, deliberately including phrases that had caused disagreement in round one. Agreement improved on the three targeted items, with AC1 values of 0.80, 0.80, and 0.62. But a fourth item, assessing whether the student accepted the patient’s preference about discussing weight, dropped from 0.83 to 0.52, possibly because the character’s new reticence changed the conversational dynamics. With only 10 cases, the confidence intervals were extremely wide, and the authors explicitly caution that this round should be interpreted as exploratory rather than as a test of independent observations, since the same authors who revised the rubric also generated the test interactions.
Phase two moved into the real world of student education. As part of a larger one-arm weight sensitivity intervention embedded in an interprofessional education curriculum, 78 medical students were invited to use the AI simulation before and after completing four e-modules on weight bias in health care. Sixty students participated, producing 122 total interactions; 56 students completed two encounters each. A random sample of 30 transcripts was scored by the same blinded expert reviewer, and AI-human agreement was calculated again. This time, seven items showed very good agreement, three showed good agreement, and one remained in the moderate range. The item asking whether students asked if weight loss was a priority—a concrete, content-specific behavior—climbed to an AC1 of 0.89 after prompt engineering. But the two items requiring broader contextual reading, offering support or partnership and accepting the patient’s preference about discussing weight, settled at 0.66 and 0.64, while the patient-preference item stayed stubbornly moderate at 0.42, with the AI still crediting 66.7 percent of transcripts versus the human rater’s 50 percent even after the prompts were made more restrictive.
The pattern across all rounds points to a clear dividing line. Items requiring recognition of specific, concrete behaviors—asking about weight management history, asking whether weight loss was a priority—showed consistently high AI-human agreement. Items requiring broader contextual interpretation or sensitivity to patient preferences lagged behind. The authors argue this shows AI does not struggle uniformly; its limitations concentrate in items that depend on interpersonal cues, contextual understanding, or small nuances in conversation. For educators, that means identifying which competencies AI can reliably assess and which still demand human judgment. For items with persistently poor agreement, the authors suggest a different framing altogether: rather than treating them as clear-cut scoring metrics, educators might use AI feedback as a starting point for group discussion and learner reflection, preserving the role of human judgment while letting AI handle the informational, surface-level feedback it does well.
Student reactions offered a cautiously encouraging coda. Of the 60 students who used the platform, 28 percent mentioned the AI in their voluntary open-ended feedback, and most of those comments—59 percent—were positive, with several students reporting that the simulation contributed to their learning. The critiques, from 41 percent of commenters, included wanting more instructions, wanting a variety of simulated patients rather than a single character, technical difficulties, and finding the feedback or patient unrealistic. The authors note that low realism can undermine the effectiveness of AI simulations, though patient realism can itself be adjusted through prompt engineering—an avenue this study did not explore. They also flag their limitations candidly: small samples with wide confidence intervals, potential bias from authors testing their own rubric revisions, differing competency levels between the calibration and student phases, and the specificity of both the platform and the weight sensitivity rubric, which includes subtle dimensions that challenge even human raters. Their conclusion is measured but clear: AI scoring in simulation-based education demands iterative prompt engineering and active educator guidance, and future work should compare student learning outcomes in AI simulations against traditional ones, including whether the skills transfer to clinical practice.
Subject of Research: AI-human interrater reliability in AI-based simulation assessment of medical students' weight bias communication skills
Article Title: How does AI-generated assessment compare to expert assessment in simulation-based experiences: Analyzing AI-human interrater reliability
Article References: Brace, M., Furman, G., Diewald, L., Shenkman, R., Felgoise, S., & Oliver, T. (2026). How does AI-generated assessment compare to expert assessment in simulation-based experiences: Analyzing AI-human interrater reliability. Advances in Simulation, 11(1), Article 70. https://doi.org/10.1186/s41077-026-00464-5
Image Credits: AI Generated
DOI: 10.1186/s41077-026-00464-5
Keywords: artificial intelligence, simulation-based education, medical students, interrater reliability, prompt engineering, weight bias, SimConverse, clinical communication, health professions education, automated feedback, Gwet's AC1, standardized patients
News Source: Courtney Benton. (October 28, 2026). AI Graded Medical Students Like Experts—Until the Rubric Got Subtle. Scienmag.
