Cross-Validation Exposes Inflated Effect Size Estimates in Brain Association Studies

Applying cross-validation in mass-univariate voxel-based neuroimaging exposes substantially inflated effect size estimates in small samples, according to an evaluation published in Translational Psychiatry. Evaluating body mass index as a predictor for gray matter density across 3,401 magnetic resonance imaging scans from a psychiatric cohort, investigators found that small sample sizes without cross-validation produced severe effect size overestimations that failed to generalize to novel data.

In psychiatric neuroimaging research, brain-wide association effects are commonly overestimated, which contributes to poor replicability, limited generalizability, and compromised translation into clinical utility. Although cross-validation is not standardly employed in univariate neuroimaging analyses, the investigative team evaluated its implementation to systematically determine whether it could serve as a counteracting measure against inflated statistical estimates.

To assess the impact of cross-validation, the researchers analyzed magnetic resonance imaging data across different brain localizations and varied sample sizes with a spectrum of underlying pooled effect sizes. In the complete cohort of 3,401 scans, body mass index displayed a negative association with gray matter density, reaching a maximum underlying partial explained variance of R²p = 0.036.

When cross-validation was omitted, effect size estimates escalated sharply in smaller cohorts, reaching values as high as R²p = 0.535. Furthermore, in samples of n = 25 where the underlying condition was a null-effect, estimated effect sizes nonetheless reached up to R²p = 0.429 without cross-validation. Implementing a cross-validation framework demonstrated that these pronounced small-sample associations did not generalize when tested on unseen data.

The analysis established that a minimum sample size of n = 100 was necessary for the largest brain-wide association effects to begin generalizing, whereas detecting smaller effects required cohorts of at least n = 400. In contrast, underlying null effects never attained this point of initial generalizability, failing to generalize even within the full cohort of 3,401 scans. The authors concluded that cross-validation effectively highlights overestimations in small datasets to address replicability deficits, providing open-source code for practical implementation.

Subject of Research: Cross-validation of effect sizes in brain-wide association studies

Article Title: Counteracting the overestimation of effect size in mass-univariate brain-wide association studies using cross-validation

Article References: Goltermann, J., Winter, N. R., Gruber, M., Fisch, L., Richter, M., Grotegerd, D., Dohm, K., Meinert, S., Leehr, E. J., Böhnlein, J., Kraus, A., Thiel, K., Winter, A., Flinkenflügel, K., Leenings, R., Barkhau, C., Ernsting, J., Berger, K., Minnerup, H., … Dannlowski, U. (2026). Counteracting the overestimation of effect size in mass-univariate brain-wide association studies using cross-validation. Translational Psychiatry, 16(1), Article 612. https://doi.org/10.1038/s41398-026-04529-x

Image Credits: AI Generated

DOI: 10.1038/s41398-026-04529-x

Keywords: brain-wide association studies, neuroimaging, cross-validation, effect size, magnetic resonance imaging, gray matter density, sample size, replicability, Counteracting, overestimation, effect, size