Common Shrinkage Formula for Prediction Models Is Biased, Simulation Study Finds

A widely used mathematical shortcut for correcting overfitting in clinical prediction models is systematically biased, according to a large simulation study published in Diagnostic and Prognostic Research. The heuristic shrinkage factor proposed by van Houwelingen and Le Cessie, a closed-form formula applied to unpenalised logistic regression models, overestimates the optimal amount of shrinkage when the model’s discrimination is high and underestimates it when discrimination is low.

Overfitting is a central problem when developing prediction models. A logistic regression model fitted to a finite dataset tends to produce predictions that are too extreme in new data from the same population, with estimated risks pushed too high for some individuals and too low for others. One common remedy is to fit a standard maximum likelihood model and then multiply all predictor effect estimates by a global shrinkage factor, pulling predictions back toward the mean. The optimal value of this factor is the one that yields a calibration slope of one when the adjusted model is validated in the target population. When a model is being developed, this optimal value is unknown and must be estimated, and the accuracy of that estimate determines whether the correction actually works.

The heuristic shrinkage factor offers a simple answer: it equals one minus the number of predictor parameters divided by the model’s likelihood ratio statistic. Because it requires only quantities available at model fitting, it has been embedded in widely adopted minimum sample size criteria for developing logistic and Cox prediction models, where analysts target a heuristic shrinkage of 0.9 or above to signal low overfitting. The bootstrap alternative, by contrast, resamples the development dataset with replacement many times, fits the model in each resample, calculates the calibration slope of each bootstrap model in the original data, and averages those slopes. It can only be computed after a model has been fitted, which is why sample size formulas have relied on the closed-form heuristic instead.

To test how well each estimator recovers the true optimal shrinkage, the researchers ran an extensive simulation built on the ADEMP framework. They randomly generated 15,000 scenarios with correlated predictors and another 15,000 with uncorrelated predictors, varying the number of measured and unmeasured predictors between one and thirty, regression coefficients drawn uniformly between minus 0.5 and 0.5, outcome proportions between 0.05 and 0.95, and development sample sizes between 100 and 5000. For each scenario they first fitted a model in a validation cohort of one million individuals to establish the population-level C-statistic and Cox-Snell R-squared, then repeated the development process 250 times, calculating the heuristic estimate, the bootstrap estimate using 200 resamples, and the true optimal shrinkage obtained from the calibration slope in the giant validation cohort.

The results revealed a clear directional pattern tied to discrimination. In scenarios where the population C-statistic fell between 0.8 and 0.85, the 95th percentile range of the heuristic estimator’s bias ran from 0.005 to 0.387, meaning it substantially overestimated the optimal shrinkage. Where the C-statistic fell between 0.6 and 0.65, the corresponding range was minus 0.580 to minus 0.007, a substantial underestimate. The magnitude of the bias grew as the C-statistic moved toward either 0.5 or 1, and shrank as sample size increased and the optimal shrinkage approached 1. Across all scenarios the mean magnitude of bias was 0.077 for the heuristic estimator, with a median of 0.032. Supplementary analyses using a data generating mechanism that decoupled the number of predictors from the C-statistic confirmed that the bias was driven by model discrimination rather than predictor count.

The bootstrap estimator performed markedly better. Its mean magnitude of bias across all scenarios was 0.024, with a median of just 0.007, and its bias was far less dependent on the C-statistic, with a median close to zero across all discrimination levels. Performance only dropped considerably when the true optimal shrinkage was very small, below 0.3, a regime that adequate sample size criteria are designed to avoid. Under the alternative data generating mechanism the bootstrap estimate showed a mild relationship with the C-statistic, slightly underestimating optimal shrinkage at high discrimination and overestimating it at low values, but the magnitude remained very small unless a large amount of shrinkage was required.

The authors are careful to stress that these are average results. At the level of an individual dataset, both estimators showed considerable variability, particularly in small samples, so even bootstrapping provides no guarantee of correctly correcting overfitting in any single study. In instability plots drawn from six scenarios with sample sizes near 250 to 4000, the variability of the bootstrap estimate was consistently smaller than the variability of the optimal shrinkage itself, meaning the estimator will not necessarily land on the required value even in scenarios where it is unbiased on average. The researchers also removed extreme scenarios, such as those with negative shrinkage estimates or non-converging bootstrap procedures, which they judged unrepresentative of realistic practice.

The second simulation examined the downstream consequences for sample size planning. Across 25,000 scenarios, the team compared the original Riley et al. formula, which targets a heuristic shrinkage of 0.9, with a simulation-based adaptation proposed by Pavlou et al. that iteratively simulates development and validation cohorts until the expected shrinkage reaches the target. Models developed at the simulation-based sample size achieved a mean optimal shrinkage much closer to 0.9, with a mean magnitude of bias of 0.004 versus 0.041 for the formula-based size. The 95th percentile range of mean optimal shrinkage was 0.814 to 0.985 for the original formula compared with a far tighter 0.882 to 0.921 for the simulation approach. The formula-based size fell short of the target on average when the C-statistic exceeded roughly 0.73 and overshot it below that threshold, whereas the simulation-based size stayed near 0.9 regardless of discrimination.

Applied examples illustrated the practical stakes. In a critical care example using the MIMIC-III dataset to predict acute kidney injury within 24 hours of ICU admission, based on 20,413 patients of whom 3,528 experienced the event, the heuristic and bootstrap estimates diverged considerably for development samples below 500, suggesting the heuristic value was likely too low in those settings. In a sample size example for a pre-eclampsia model with 13 predictors, an outcome proportion of 0.05 and a C-statistic of 0.71, the two approaches agreed closely, giving 4114 versus 3820 participants. But when extra predictors raised the anticipated C-statistic to 0.85, the simulation-based requirement of 2985 was 38 percent larger than the formula’s 2157, and when only five predictors were available with a C-statistic of 0.6, the simulation-based requirement of 2280 was 30 percent smaller than the formula’s 3236.

The authors conclude that the heuristic shrinkage factor is often a poor estimator of the optimal global shrinkage factor and recommend the bootstrap estimate whenever global shrinkage is applied after model development, echoing existing guidance. They likewise recommend simulation-based sample size calculation over formulas targeting the heuristic value, while cautioning that care is needed when the anticipated C-statistic is very low, below about 0.6, where the simulation approach occasionally failed to converge. They also emphasise that minimum sample size criteria are absolute minimums: even at the simulation-based size, considerable variation in optimal shrinkage persisted across repeated data collections, so larger samples remain the most reliable protection against overfitting. The threshold of 0.73 marking the switch from underestimation to overestimation, they note, should be tested in further simulations with different data generating mechanisms.

Subject of Research: Bias of heuristic versus bootstrap shrinkage factor estimators in logistic regression prediction models, assessed by simulation

Article Title: Agreement between heuristic shrinkage factor and optimal shrinkage factors in logistic regression for risk prediction: a simulation study across different sample sizes and settings

Article References: Pate, A., Martin, G. P., & Riley, R. D. (2026). Agreement between heuristic shrinkage factor and optimal shrinkage factors in logistic regression for risk prediction: a simulation study across different sample sizes and settings. Diagnostic and Prognostic Research, 10(1), Article 15. https://doi.org/10.1186/s41512-026-00222-1

Image Credits: AI Generated

DOI: 10.1186/s41512-026-00222-1

Keywords: shrinkage factor, logistic regression, prediction models, overfitting, simulation study, bootstrap, sample size calculation, calibration slope, C-statistic, clinical prediction, van Houwelingen, Riley criteria