Selecting Training Data via SHAP Scores Boosts Classification Accuracy in Clinical Models

Researchers evaluating predictive modeling in medicine face persistent difficulties with measurement errors, uninformative variables, and heterogeneous patient populations that often undermine model generalizability. A newly reported investigation examines SHAP-Enhanced Predictive Modeling, an observation-selection framework designed to address these challenges by curating training data through aggregated feature attributions. Tested across four clinical benchmark datasets and five machine learning classifiers, the framework demonstrated that models trained on pruned data subsets could frequently surpass standard full-sample training. Specifically, the method achieved higher mean F1 scores than conventional full-data modeling in eighteen of twenty evaluated dataset-classifier configurations, while also outperforming an independently optimized random-selection control in sixteen instances. These gains were realized while retaining between 42.1 percent and 94.4 percent of training observations. Because expanding sample size in healthcare data does not automatically ensure reliability, the findings clarify that training observation composition plays a decisive role in shaping model behavior across diverse clinical prediction tasks.

Traditional efforts to enhance clinical machine learning typically focus on model architecture, predictor dimensionality reduction, ensemble construction, or mathematical regularization that penalizes excessive complexity. In contrast, the newly evaluated framework implements a data-centric paradigm prioritizing individual training observations according to estimated relevance to the learning algorithm. To achieve this, the authors repurposed Shapley Additive Explanations, a technique originally established in explainable artificial intelligence for interpreting local model decisions rather than estimating data valuation. Within the proposed framework, each training instance receives an importance score calculated as the mean absolute SHAP value across all input predictors. Summing absolute values captures aggregate attribution magnitude without distinguishing between positive and negative contributions to the predicted outcome. The authors emphasize that these importance metrics operate strictly as a ranking heuristic rather than as formal estimates of an observation’s marginal predictive utility, meaning low-scoring patient records cannot simply be dismissed as uninformative or noisy data.

The operational pipeline of the framework comprises four distinct phases structured within a nested cross-validation design to prevent performance leakage. First, a baseline classifier is fitted to the outer training partition with hyperparameter tuning conducted through stratified inner cross-validation. Preprocessing routines, including scaling and missing data imputation, are derived solely from training folds and applied to held-out validation partitions. Following baseline training, observation-level scores are computed using algorithm-appropriate SHAP estimators, including linear estimators for logistic regression, kernel-based estimators for support vector machines, tree-specific implementations for decision ensembles, and native contribution routines for gradient-boosted trees. Candidate training subsets are formed by retaining records whose importance scores meet or exceed quantile thresholds for tested retention fractions. To guarantee valid binary classification, missing outcome classes are supplemented based on score hierarchy. Inner validation folds select the retention proportion producing the highest mean F1 score, after which the model is refitted and evaluated on outer test data.

To determine whether performance differences stemmed from feature-attribution ranking rather than training-set reduction alone, the researchers established rigorous experimental controls. A randomized selection procedure, designated as the random control, was built using identical classifier configurations, hyperparameter spaces, inner cross-validation splits, and candidate retention proportions. For every candidate fraction, three independent stratified random subsets were sampled to preserve original class proportions and guarantee both outcome classes were present. The validation F1 scores across these three random repetitions were averaged to select the optimal retention proportion for the control pipeline. Because retention proportions were optimized independently for both strategies, final retained sample sizes could differ between attribution-guided and randomized methods. Consequently, the study evaluated complete selection procedures rather than isolating attribution ranking at an identical fixed subset size. In parallel, a traditional machine learning baseline trained models on all available outer-fold instances, providing a direct reference for assessing whether selective observation pruning delivers advantages over complete dataset retention.

Empirical validation was conducted using four publicly available clinical datasets exhibiting varied sample sizes, class balances, and feature distributions. The cervical cancer behavior risk dataset comprised 72 patient records, including 21 positive cases and 51 negative cases tracking behavioral risk indicators. The breast cancer recurrence cohort contained 286 instances, encompassing 85 recurrence events and 201 non-recurrence cases described by pathological and clinical characteristics. The Pima Indians diabetes dataset provided 768 patient entries with eight clinical measurements, containing 268 positive and 500 negative diagnoses. Finally, Haberman’s survival dataset included 306 surgical patients, documenting 81 individuals who died within five years and 225 who survived longer. Preprocessing addressed specific dataset characteristics, such as replacing physiologically implausible zero measurements in the diabetes cohort using iterative multivariate linear regression imputation. Categorical predictors in the breast cancer dataset were imputed, numerically encoded, and standardized. Five classification algorithms representing linear, kernel, bagging, and boosting paradigms were applied: logistic regression, support vector machines, random forests, gradient boosting machines, and XGBoost.

Predictive capability across the twenty clinical benchmark tasks was evaluated through outer-fold mean F1 scores alongside balanced accuracy using a fixed 0.50 decision threshold. The attribution-guided framework achieved the highest mean F1 score in 16 of the 20 tasks, surpassing the full-data baseline in 18 configurations. Full-data training achieved superior F1 scores in only two scenarios: support vector machines on cervical cancer and logistic regression on diabetes. In both breast cancer recurrence and Haberman survival cohorts, the selection framework outperformed full-data training across every classifier. Statistical significance was verified using the Demšar multi-dataset testing framework. A global Friedman test confirmed significant performance divergence across the three modeling strategies, yielding a chi-square statistic of 16.127 with two degrees of freedom and a Kendall concordance coefficient of 0.403. Pairwise exact two-sided Wilcoxon signed-rank tests with Holm step-down adjustments confirmed that the attribution-guided framework significantly outperformed both full-data training and randomized pruning, whereas differences between randomized pruning and full-data training were not statistically significant.

Despite demonstrated improvements in threshold-based classification accuracy, the study revealed critical drawbacks regarding the calibration of predicted probabilities. Across the twenty benchmark combinations, the attribution-guided framework yielded the highest overall mean balanced accuracy, but produced higher expected calibration error in 17 tasks and higher Brier scores in 15 tasks compared with full-data training. Expected calibration error was assessed across ten equally spaced bins to measure discrepancies between model risk estimates and empirical event frequencies, while Brier scores quantified squared differences between predicted probabilities and observed binary outcomes. Because training-subset selection explicitly maximized classification F1 scores, optimization prioritized decision boundaries over reliable probability outputs. The researchers noted that elevating F1 scores does not inherently generate dependable risk estimates, underscoring that future clinical implementations must incorporate calibration-aware objective functions or dedicated probability recalibration.

To assess computational and predictive trade-offs against formal data valuation, the investigators benchmarked the framework against Truncated Monte Carlo Data Shapley using XGBoost across all four clinical datasets. The common nested evaluation protocol allocated 80 percent of training observations to candidate pools and 20 percent to valuation sets. Monte Carlo valuation approximated marginal contributions across 100 permutations, truncating iterations when utility remained within an absolute tolerance of 0.01 for five consecutive additions. Monte Carlo valuation achieved higher mean F1 scores and balanced accuracy across all four datasets, whereas the attribution-guided framework recorded lower calibration errors and Brier scores. None of the four dataset-level performance differences achieved statistical significance under two-sided exact Wilcoxon testing, where four paired samples yielded a minimum attainable p-value of 0.125. However, pipeline runtime ratios of Monte Carlo valuation relative to the attribution framework ranged from 64.8 to 1216.5, illustrating the substantial computational advantage of single-pass attribution ranking over permutation-based coalition modeling.

A comprehensive simulation study evaluated the framework under controlled conditions across 20 synthetic scenarios varying training sizes from 100 to 5,000 and outcome prevalences from 10 percent to 35 percent. Generating 2,000 datasets with informative variables, uninformative noise, unobserved population subgroups, and heavy-tailed measurement errors, the experiments evaluated held-out test groups of 5,000 observations. The attribution-guided framework achieved the highest mean F1 score across all 20 scenarios on the full test group, with performance advantages most pronounced at lower outcome prevalences within larger sample sizes. Stratifying test sets into high-attribution and low-attribution groups revealed that the selection framework retained its F1 advantage across both partitions, producing its largest performance gains in the low-attribution test cohort. However, full-data training consistently maintained superior probability calibration and lower Brier scores across simulation settings. These findings confirmed that attribution-guided sample pruning improves binary classification under severe imbalance and noise, but systematically exacerbates probability distortion across both common and uncommon patient profiles.

The investigators underscored several important limitations that must temper translation of attribution-guided training into healthcare environments. Benchmark datasets were modest in scale, particularly the cervical cancer cohort with only 72 observations, and internal nested cross-validation cannot substitute for external validation on independent clinical cohorts. Furthermore, attribution metrics reflect initial baseline model fits, creating risks of discarding atypical clinical presentations that represent genuine, vulnerable patient subpopulations rather than measurement noise. The clinical classifier comparisons were also not completely independent due to shared outer partitions. Moving forward, the authors recommend conducting prospective external validations, examining ranking stability across data splits, evaluating fixed-size subset comparisons, assessing performance within pre-specified clinical subgroups, and incorporating probability-aware optimization criteria. The analysis code, experimental configurations, and verification scripts have been made publicly available on GitHub to support independent replication and methodological extension before such data-centric observation selection approaches can be considered for real-world clinical deployment.

Subject of Research: Data-centric observation selection for clinical machine learning models

Article Title: SHAP enhanced predictive modeling for clinical machine learning

Article References: Amponsah, J. K., Thompson, J. A., & Ellis, S. D. (2026). SHAP enhanced predictive modeling for clinical machine learning. Discover Health Systems, 5(1), Article 94. https://doi.org/10.1007/s44250-026-00440-w

Image Credits: AI Generated

DOI: 10.1007/s44250-026-00440-w

Keywords: clinical machine learning, SHAP, data-centric AI, predictive modeling, training data selection, probability calibration, XGBoost, feature attribution, enhanced, predictive, modeling, clinical