Trained students can match radiologists in knee ultrasound ratings, study finds

Trained university students, working in groups and guided by a professional sonographer, can rate knee recess distension on ultrasound images with accuracy approaching that of an expert radiologist, according to a study published in Discover Imaging. The research, led by R. Fallahpour and colleagues at the University of Toronto network with co-authors including Pascal N. Tyrrell, tested whether students could stand in for scarce and costly expert raters when large medical imaging datasets need to be labeled. The answer, the authors report, is a qualified yes: with the right training and enough raters averaged together, student segmentations of joint effusions came close to expert benchmarks, though the most meticulous expert precision remained out of reach.

The motivation is practical. In medical imaging research, datasets are often built from measurements, classifications, or diagnoses made by radiologists acting as raters, and agreement analysis of those ratings validates the reliability and consistency of the data. Expert radiologists deliver high precision, but their limited availability and high cost can restrict scalability for large datasets. The study focused on a clinically relevant scalar quantity: the estimated area of joint effusion, delineated in the suprapatellar recess of adult patients with clinically evident knee recess distension. Effusion size can help track inflammatory disease activity, for example in rheumatoid arthritis, and can support the training of artificial intelligence models designed to detect or quantify effusions. The authors stress that segmentation is not required for routine clinical decision-making; consistent delineation is valuable mainly for building automated tools and for longitudinal research assessment.

The base dataset was substantial. It comprised 11,676 images from 3,710 knee examinations, with a recess distension positivity rate of roughly 75 percent, drawn from adult patients with a mean age of about 50 who underwent ultrasounds between 2010 and 2018. The data were anonymized, encrypted, and transferred to the lab archive with all patient demographics excluded. Examination reports were processed using natural language processing to identify positive and negative cases, and an expert sonographer with 32 years of experience validated the presence or absence of recess distension. From this pool, a professional sonographer selected ultrasound images of 25 patients diagnosed with recess distension, one image per patient, for segmentation by both students and expert radiologists.

Ten university students with limited prior experience were recruited and trained in three stages. A sonographer provided foundational knowledge on identifying recess distension in adult knee ultrasound images. A graduate student of medical imaging then taught segmentation using the Label Studio software. Finally, students learned the submission protocol: three segmentations for each assigned image. Once they met the required standards based on the sonographer’s evaluation, the students rated each of the 25 ultrasound images three times in random order. An expert radiologist segmented the same images to establish a benchmark, and the area of the segmented region, the masked surface, was used for both student and expert comparisons.

Performance was measured with three statistics. The coefficient of variation, or CV, captured each student’s variability across the three replicate measurements per image over the 25 patients. The intraclass correlation coefficient ICC(2,1) reflected agreement between student ratings and the expert’s ratings. Root mean square error, or RMSE, quantified deviations between student and expert segmentations. The results showed ICC values ranging from 0.76 to 0.95. Most students achieved ICC values above 0.80, indicating good to excellent agreement with the expert. One student showed moderate agreement with an ICC of 0.76 and higher RMSE values, reflecting variability in performance. Higher ICC values typically corresponded to lower RMSE, underscoring that consistency in ratings goes hand in hand with reduced error.

The heart of the study was a statistical simulation asking how many inexperienced student raters are needed to reach accuracy comparable to experts. Ground truth segmentation areas were simulated from a normal distribution, and errors for students and experts were generated from their respective CVs by adding normally distributed noise to fixed ground truth values. Student ratings were averaged within subsets ranging from 1 to 10 raters, and RMSE against ground truth was computed over 1,000 repetitions. Three student variability levels were tested: low (CV 0.55), medium (0.7), and high (0.9). Expert benchmarks came from two sources: CV values of 0.2 and 0.3 drawn from the literature, representing meticulous experts, and the CV of 0.7 observed in the study’s own collected data.

The simulation showed that for all student CV levels, RMSE decreases as the number of raters increases, demonstrating the compensatory effect of aggregation, though the improvement eventually reaches a saturation point. The low CV group consistently outperformed the medium and high groups, achieving an RMSE below the benchmark set by Expert CV 0.7. Roughly three low-variability student raters were sufficient to meet the RMSE benchmark of Expert CV 0.3, and about eight were needed to approach the even lower RMSE of Expert CV 0.2. The medium CV group required approximately six raters to meet the Expert CV 0.3 benchmark but failed to reach Expert CV 0.2 even with all ten raters. The high CV group needed more than nine raters to reach the Expert CV 0.3 benchmark and could not match Expert CV 0.2 at any aggregation size considered.

A clinical analysis of the actual data echoed the simulation. RMSE was highest, around 10,000, when only one student was involved, then decreased sharply with noticeable improvements between two and four students. The rate of improvement slowed after four student raters, suggesting that once an optimized number of raters is employed, accuracy stabilizes. With ten student raters, all CV groups achieved RMSE values approaching or surpassing the Expert CV 0.7 benchmark derived from the study’s own data collection. The authors note that the relatively high expert CV of 0.7 highlights the inherent difficulty of the task: joint effusions have irregular, low-contrast boundaries that contribute to variability even among experienced clinicians, so the observed expert disagreement likely reflects the imaging modality and task ambiguity rather than individual rater performance.

The implications are framed carefully. The authors emphasize that this task does not represent a standard-of-care clinical measurement but a research annotation effort. The goal was not to replace experts in clinical decision-making but to evaluate whether trained student raters could reliably contribute to research workflows, such as dataset labeling for AI development. From a cost perspective, the findings are significant: for tasks where the expert CV benchmark is relatively high, groups of student raters can effectively replace experts with adequate training and aggregation, reducing reliance on expensive expert evaluations. In scenarios requiring the highest precision, such as an Expert CV of 0.2, expert raters remain indispensable. The authors also observe that small variations in segmented effusion area may not meaningfully impact diagnostic or treatment decisions when effusion presence is binary or semi-quantitative, though precise volume quantification or follow-up comparisons still require expert interpretation.

Limitations are acknowledged. Area alone may not capture all aspects of segmentation quality, such as shape or boundary accuracy, and the datasets analyzed are not publicly available due to confidentiality, though they are available from the corresponding author on reasonable request. The study examined a single task, knee recess distension in adult ultrasound images, and the authors call for future research on generalizability across different diseases and modalities such as MRI and CT, including pathologies like tumors, organ lesions, and inflammatory markers. Still, the core message stands: trained student raters, with structured training and optimized aggregation, offer a cost-effective and scalable alternative for resource-limited settings in medical imaging research, bridging gaps in accessibility while maintaining acceptable accuracy for research applications.

Subject of Research: Feasibility of using trained student raters instead of expert radiologists to assess knee recess distension in adult ultrasound images

Article Title: Replacing experts with student raters for assessing knee recess distension in ultrasound images of adult patients

Article References: Fallahpour, R., Radigan, L. H., Mendez, M., Nashnoush, M., & Tyrrell, P. N. (2025). Replacing experts with student raters for assessing knee recess distension in ultrasound images of adult patients. Discover Imaging, 2(1), Article 12. https://doi.org/10.1007/s44352-025-00015-0

Image Credits: AI Generated

DOI: 10.1007/s44352-025-00015-0

Keywords: ultrasound, knee recess distension, medical imaging, student raters, segmentation, intraclass correlation coefficient, root mean square error, coefficient of variation, aggregation, joint effusion, radiology, AI training data