Clinical Prediction Stability Depends on Algorithms and Tuning Beyond Sample Size

The stability of clinical prediction models depends not only on sample size but also on the chosen modelling algorithm and hyperparameter tuning, according to an empirical study published in BMC Medical Research Methodology. In clinical practice, models that yield unstable or unreliable predictions can ultimately lead to improper clinical decisions and cause harm to patients. While previous research demonstrated that larger sample sizes deliver increased stability, the influence of other modelling choices has remained unclear. To address this question, researchers Phichayut Phinyo, Daniel Stahl, and Ewan Carr evaluated three modifiable factors affecting prediction stability: sample size, modelling algorithms across parametric regression and machine learning, and hyperparameter tuning.

The researchers conducted their investigation using the previously published HOPE dataset to develop a prediction model for anxiety at palliative care admission. The modelling process incorporated 34 candidate predictors available within the dataset. To investigate the specific role of sample size on stability, the team generated seven datasets of varying sizes through random sampling from the clinical data.

In total, the study evaluated 21 distinct modelling scenarios by pairing the seven datasets of varying sizes with three modelling approaches: LASSO logistic regression, random forest, and XGBoost. The models were tuned using cross-validation and evaluated for discrimination via the area under the receiver operating characteristic curve (AUC) as well as calibration. The authors assessed prediction stability through a bootstrap-based evaluation framework that tracked the mean absolute prediction error (MAPE) alongside related instability plots.

Within the clinical dataset, larger sample sizes were associated with lower average MAPE and a higher proportion of predictions achieving a MAPE of 5% or less for LASSO logistic regression. The authors observed limited additional changes in these stability metrics beyond the reference sample size of 3,875 individuals. Furthermore, across increasing sample sizes, LASSO models demonstrated lower AUC optimism and produced calibration slopes that were closer to 1.

By comparison, the machine learning algorithms random forest and XGBoost demonstrated higher discrimination than LASSO in certain scenarios, but they exhibited higher average MAPE, a lower proportion of predictions with MAPE of 5% or less, and calibration slopes further from 1 across sample sizes. Tuning conservative hyperparameters, such as configuring shallower trees and slower learning rates, was associated with lower MAPE but resulted in slightly lower discrimination. The authors concluded that evaluating stability alongside marginal gains in accuracy may be critical for machine learning models, noting that future research should examine alternative validation strategies and optimal tuning practices to guide robust clinical prediction model development.

Subject of Research: Prediction stability in clinical prediction model development

Article Title: The influence of sample size, modelling algorithms, and hyperparameters on bootstrap-based prediction stability in clinical prediction model development: An empirical study using the HOPE dataset

Article References: Phinyo, P., Stahl, D., & Carr, E. (2026). The influence of sample size, modelling algorithms, and hyperparameters on bootstrap-based prediction stability in clinical prediction model development: An empirical study using the HOPE dataset. BMC Medical Research Methodology. https://doi.org/10.1186/s12874-026-02987-x

Image Credits: AI Generated

DOI: 10.1186/s12874-026-02987-x

Keywords: clinical prediction models, prediction stability, bootstrap evaluation, machine learning, sample size, hyperparameter tuning, LASSO regression, influence, sample, size, modelling, algorithms