Machine-translated radiology reports fluent yet error-prone, expert review essential

Machine translation has become an attractive shortcut for radiology departments that must move clinical reports across language boundaries. Teleradiology, international referral pathways, and cross-border care all create situations in which a report written in one language must be understood by clinicians working in another. A new retrospective study from University Hospital Heidelberg, published in European Radiology Experimental, tested how safely this can be done in practice, and its central message is cautionary: fluent output is not the same as safe output.

Researchers selected 310 de-identified German radiology reports from June and July 2025, drawn by stratified random sampling across report-length strata. The sample spanned ten imaging modalities and multiple body regions and included 37 structured and 273 primarily free-text reports. Report length ranged from 34 to 1,529 words, with a median of 136 words. Dates, patient names, and other directly identifying information were removed before translation, and no abbreviation expansion or terminology normalization was performed. All reports were translated from German to English using the DeepL Application Programming Interface on January 27, 2026, with default settings and no custom glossary.

Eight radiologists—two attending radiologists and six residents with a mean of three years of experience, all working clinically in German—compared each German source report with its English translation. Each reader assigned an overall quality score on a five-point scale, from unusable to excellent, and directly corrected wording they considered inaccurate or in need of revision. Thirty reports were reviewed by all eight readers, while 280 additional reports were assigned for single-reader evaluation, producing 520 expert evaluations in total. Readers were blinded to each other’s assessments and to all automated metrics and model outputs.

The headline result was double-edged. Most translations were rated good or excellent—339 of 520 evaluations, or 65.2 percent—and the mean quality score was 3.73 out of 5.0. Yet expert review still uncovered clinically important mistranslations involving diagnosis, anatomy, and radiologic terminology. Representative examples included the German term “Mehrsklerosierung” being rendered as “multiple sclerosis,” an interphalangeal joint being mistranslated as a metatarsophalangeal joint, and white matter changes being translated as medullary changes. In shared reports, readers often corrected the same source error even when their individual wording differed.

Perhaps the most striking methodological finding concerned how inconsistently radiologists rated the same translations. Agreement on the ordinal quality scores was very low, with a Krippendorff alpha of 0.078 (p = 0.006) and an intraclass correlation coefficient of 0.096, indicating substantial variability in how readers assigned severity to identical reports. By contrast, agreement on the actual corrections was high, with a mean pairwise BLEU score of 83.0 (p < 0.001). In other words, experts disagreed about how bad a translation was overall but agreed closely on which specific wording needed to change—a distinction the authors argue matters greatly for quality assurance, because a correction directly identifies the text requiring review.

The team then asked whether automated methods could flag which reports needed human review. Using a deliberately conservative, safety-oriented endpoint—classifying a report as review-positive if even one of the eight readers gave it a score of 2 or lower—18 of the 30 shared reports were review-positive. Under this primary screen, no evaluated automated method significantly discriminated reports requiring review, with area under the receiver operating characteristic curve values ranging from 0.514 to 0.653 and p-values from 0.100 to 0.920. The evaluated methods included BLEU, chrF++, BERTScore, COMET quality estimation, and two reference-free large language model judges, Claude Sonnet 4 and GPT-5.1.

In a stricter sensitivity analysis requiring at least two readers to rate a report 2 or lower, eight of 30 reports were review-positive. Conventional metrics and mean edit rate again showed limited discrimination, with AUCs between 0.500 and 0.636. The two large language model judges, however, showed moderate discrimination in this setting: Claude Sonnet 4 achieved an AUC of 0.790 (95 percent CI 0.597–0.956; p = 0.005) and GPT-5.1 an AUC of 0.773 (95 percent CI 0.588–0.933; p = 0.015). Even so, these judges operate at the report level and do not indicate which specific wording should be checked.

To address localization, the researchers built a correction-based segment-level edit-prediction model. German source reports and their translations were partitioned into overlapping 16-token windows with 50 percent overlap, plus 32-token windows for cross-scale averaging. Paired German and English segments were embedded with multilingual E5 and concatenated into a 4096-dimensional feature vector, and a multilayer perceptron was trained to predict whether a segment would be edited by a radiologist. Training combined supervised contrastive pretraining with clinically informed synthetic perturbations—terminology substitutions, abbreviation errors, anatomic or laterality changes, altered negation or uncertainty, numerical changes, and untranslated text—followed by fine-tuning on expert correction labels. Ill-fit negative pairs, pairing source segments with unrelated text, were added to improve robustness.

Model development used the 280 single-reader reports with fivefold cross-validation, while testing was performed exclusively on the 30 held-out shared reports, comprising 601 analyzable 16-token segments. The final model, incorporating cross-scale averaging, achieved a concordance index of 0.697 (95 percent CI 0.668–0.724; p < 0.001) and an AUC of 0.731 (95 percent CI 0.688–0.769) for segments edited by at least six of eight readers, and an AUC of 0.749 for segments edited by at least four. Predicted edit probability rose with multi-reader consensus, with a segment-level Spearman correlation of 0.498 (p < 0.001) between predicted probability and the number of correcting readers. At the report level, BERTScore showed the largest correlations with mean quality score (rho = 0.588) and number of correcting readers (rho = −0.700) but required reference translations, whereas the reference-free judges correlated more modestly with corrections.

The study also ran a secondary contextual benchmark of 51 translation-system configurations—nine proprietary LLM, 14 open-weight LLM, and 28 dedicated machine-translation configurations—against the expert-corrected DeepL outputs, using BLEU, chrF++, BERTScore, COMET, and COMET-QE. Proprietary LLMs had the highest reference-based point estimates, with GPT-5.1 achieving the highest BLEU score of 61.5, followed by Claude Sonnet 4 and Claude Opus 4, but all systems remained below the inter-reader human ceiling of 85.8. System rankings were highly stable across reader references: variance decomposition attributed 99.5 percent of BLEU variance to system quality and only 0.3 percent to the reader reference. The authors caution, however, that because the references were post-edits of DeepL output rather than independent translations, the benchmark reflects similarity to DeepL-style phrasing rather than a definitive ranking of clinical translation safety.

The authors are explicit about the limitations. The study covered a single high-resource language pair, German to English, at a single institution, and error types may differ for lower-resource languages or other reporting conventions. The multi-reader shared set was modest—30 reports and 601 segments—limiting statistical power for binary report-level screening. Crucially, both the review-trigger endpoint and the edit-prediction model were based on expert ratings and corrections, not adjudicated downstream patient outcomes; they identify text that may require expert review, not direct patient harm or diagnostic error. The study was also retrospective and did not test whether localization reduces review time or improves reviewer accuracy in live workflows.

Those caveats frame the practical takeaway. Machine-translated radiology reports were often acceptable on average, but translations containing errors requiring expert review persisted even within that fluent majority, and automated metrics optimized for average similarity missed rare but clinically consequential errors. The correction-based edit-prediction model is presented as a research-stage review-priority signal, not a deployable triage tool or autonomous safety filter. The authors argue the findings support a human-in-the-loop approach to translation oversight, in which automated systems prioritize text for expert verification rather than replace clinical judgment, and they call for prospective workflow studies to determine whether correction-based localization improves the safety and efficiency of radiology report translation before any clinical implementation.

Subject of Research: Machine translation of radiology reports and expert review of clinically important mistranslations

Article Title: Fluent but not fail-safe: machine-translated radiology reports still need expert review

Article References: Seibold, C., Leichenich, T., Stein, P., Atsiatorme, E., Kliesch, P., Sevim, G. A., Trauth, P., Kremer, A., Bischoff, A., Kauczor, H.-U., & Fink, M. A. (2026). Fluent but not fail-safe: machine-translated radiology reports still need expert review. European Radiology Experimental, 10(1), Article 158. https://doi.org/10.1186/s41747-026-00827-7

Image Credits: AI Generated

DOI: 10.1186/s41747-026-00827-7

Keywords: machine translation, radiology reports, DeepL, large language models, translation quality, clinical safety, expert review, edit prediction, teleradiology, cross-border care, quality metrics, human-in-the-loop