Medical Model Fine-Tuning Conceals Major Question-Level Accuracy Turnover

Fine-tuning medical language models can produce substantial item-level correctness turnover that remains hidden beneath minor variations in aggregate accuracy scores, according to an evaluation of a 27-billion-parameter parent model and five specialized derivatives. In medical question-answering tests, summary accuracy figures reflected the net direction of performance shifts following fine-tuning but failed to indicate the true magnitude of underlying change, masking significant redistribution in which individual items models answered correctly.

Comparative evaluations between general-purpose language models and medically specialized systems are conventionally conducted between fixed checkpoints evaluated at a single operating point. In standard benchmarking practice, these evaluations alter model scale, training data, and inference scaffolding simultaneously while leaving the fine-tuning configuration uncontrolled, obscuring which specific training choices drive the resulting performance differences.

To assess these effects under controlled conditions, researchers Adnan Agha and Eram Anwar at the United Arab Emirates University College of Medicine and Health Sciences established an experimental framework that held model scale, underlying architecture, evaluation items, decoding configurations, and scoring procedures constant. Starting from a single 27-billion-parameter parent checkpoint, the authors evaluated five fine-tuned derivatives that differed in reasoning-trace content, trace length, and training weight.

All model checkpoints were tested on the identical 2,450-item MedXpertQA-Text evaluation set. This shared benchmark allowed the researchers to conduct paired item-level performance comparisons across the parent model and each derivative, with statistical significance determined using McNemar exact tests.

Accuracy across the 2,450 test items spanned 16.86 percentage points between the parent model and its derivatives, ranging from a peak of 43.76% down to 26.90%, while spanning 15.30 percentage points among the five fine-tuned derivatives themselves. None of the fine-tuned derivatives exceeded the accuracy achieved by the original parent checkpoint. Because the fine-tuning configurations differed across several training dimensions and only one run was performed for each setup, the authors noted that these outcomes represent checkpoint-level disparities rather than the isolated effect of any single training variable, illustrating how wide performance ranges can complicate comparisons between unmatched generalist and specialist models.

Aggregate accuracy proved to be an inadequate summary of model behavioral change across the benchmark. Four fine-tuned derivatives that completed evaluation within 2.41 percentage points of the parent model nonetheless disagreed with it on 15.1% to 18.2% of test items. Even after excluding items on which either model reached the generation cap, the models still diverged on 12.8% to 14.5% of questions, while model retention rates ranged between 47.2% and 80.3%.

The study found no consistent descriptive relationship between reasoning rationale length and overall test accuracy, and no category-level retention differences remained statistically significant after multiplicity corrections were applied. The authors concluded that because aggregate accuracy can obscure extensive turnover in item-level correctness, benchmarking studies evaluating parent models alongside fine-tuned derivatives on identical question sets should directly compute and report retention, acquisition, and correctness turnover alongside overall accuracy.

The study, published in Frontiers in Digital Health within the Health Informatics section, directly addresses model evaluation standards in clinical natural language processing and machine learning. By examining how fine-tuning alters performance across 2,450 medical question-answering items, the investigation demonstrates why aggregate accuracy alone provides an incomplete benchmark for evaluating specialized large language models.

A central methodological recommendation highlighted by corresponding author Adnan Agha and co-author Eram Anwar is the necessity of reporting granular item-level dynamics whenever parent checkpoints and adapted derivatives are assessed on identical evaluation sets.

The authors also emphasize that benchmarking practices must account for these dynamics when assessing specialized medical models against general-purpose counterparts.

Subject of Research: Model evaluation and item-level correctness turnover in fine-tuned medical language models

Article Title: Small aggregate accuracy changes conceal substantial item-level correctness turnover after fine-tuning in medical question-answering

Article References: Agha, A., & Anwar, E. (2026). Small aggregate accuracy changes conceal substantial item-level correctness turnover after fine-tuning in medical question-answering. Frontiers in Digital Health, 8, Article 1956469. https://doi.org/10.3389/fdgth.2026.1956469

Image Credits: AI Generated

DOI: 10.3389/fdgth.2026.1956469

Keywords: medical question answering, fine-tuning, large language models, model evaluation, benchmarking, clinical natural language processing, machine learning, Small, aggregate, accuracy, changes, conceal