Big Data’s Big Trap: Why Huge Datasets Can Be Precisely Wrong

Biomedical science has never had more data. Databases such as ECHO, SEER-Medicare and UK Biobank, along with electronic health records, now link clinical, environmental, genomic and social information on an unprecedented scale, promising statistical power that small studies could never achieve. Yet a new Guidelines and Guidance article in PLOS Medicine by biostatistician Wonsuk Yoo of the University of Miami and John P. A. Ioannidis of Stanford University’s Meta-Research Innovation Center argues that this abundance carries a hidden cost: the very size of these datasets can manufacture convincing-looking results that are scientifically hollow. The authors dissect four major validity threats and offer a twelve-point framework for rescuing big-data inference from itself.

The first threat is what the authors call p-value inflation. Null hypothesis significance testing was designed for an era of scarce, expensive data, when random error was the dominant worry. In a large dataset, sampling variability shrinks, standard errors become tiny, and test statistics explode even when the underlying effect is biologically or clinically trivial. The result is an overabundance of very small p-values that look persuasive but correspond to effect sizes too small to matter. With big data, the authors write, the key question changes from “can we find it?” to “is it worth finding?”

Simulations in the article make the point vividly. In one set of 300 repeated two-sample t-tests per sample size, with a constant small standardized effect, p-values declined systematically as the total sample grew from 100 to 9,000, even though the effect never changed. In another simulation of 200 linear regression models with 5,000 observations each and predominantly small true coefficients, many results still yielded highly significant p-values. A multivariable simulation with ten predictors showed that at very large sample sizes, trivial true effects and spurious, confounded associations became nearly as statistically significant as genuinely meaningful ones. Statistical significance, the simulations suggest, increasingly reflects sample size and precision rather than effect magnitude, practical importance, or causal validity.

The authors identify three interlocking mechanisms. True-but-not-worth-it effects get detected because test statistics grow roughly with sample size, so doubling or tripling the number of observations can turn a borderline association into a highly significant one for the same effect size. Spurious effects get detected because real-world databases almost always contain model misspecification or leftover noise, such as unmeasured confounders or imperfect harmonization, and in huge samples the model responds to that misspecification by producing low p-values. And too many effects get pursued at once, since analysts frequently run numerous related tests across outcomes, subgroups and exposures. Multiplicity corrections solve only part of the problem, because many trivial and spurious findings survive them, and no adjustment accounts for the massive analytical onslaught of many teams attacking the same public dataset.

The second threat is residual and differential confounding across sites. Large datasets often combine data from different regions, institutions and populations, and exposures, outcomes and covariates vary systematically by site. Urban cohorts of children, for example, may face higher pollution, different socioeconomic backgrounds and different healthcare access than rural ones. When such context-specific factors affect both exposure and outcome and are not fully captured by measured covariates, site-level confounding biases the aggregate association. The problem deepens when models treat all sites as interchangeable and pool them with a single intercept, presuming uniformity in exposure-outcome relationships that rarely exists in practice.

Simulation results show how damaging this can be. In a ten-site simulation with 500 individuals per site and a common true exposure effect, site-specific models that omitted an unmeasured confounder produced estimates that varied substantially: some sites overestimated the effect, others attenuated it toward the null, and several produced estimates in the opposite direction, creating a false appearance of effect modification or poor reproducibility. A second simulation showed that as site-to-site imbalance in an unmeasured confounder grew, the pooled estimate diverged progressively from the true value despite unchanged sample size and apparent precision. Larger samples do not reduce residual confounding; they may amplify its impact through greater precision, yielding very precise but systematically wrong inferences. The authors even argue that a single pooled dataset can perform worse than the traditional route of small studies synthesized by meta-analysis, if site provenance is ignored.

The third threat arises from harmonization and measurement error. Merging variables collected under different designs, instruments and time periods introduces new layers of error: asthma may be defined by parental questionnaires in one cohort and physician diagnosis in another, and dietary data may come from 24-hour recalls at one site and food-frequency questionnaires at another. Classical random error biases coefficients toward zero, masking true relationships, while systematic errors that vary by cohort produce biases whose direction and magnitude differ by site. Because bias does not shrink with sample size, small structural errors become statistically significant as precision rises, creating an illusion of confidence in biased estimates. Most large datasets are also routinely collected data, such as insurance claims and health records, not gathered primarily for research, which compounds unrecognized measurement problems.

Harmonization asymmetry adds a further twist: variables are often reduced to the lowest common denominator so all cohorts can contribute, trading validity for inclusiveness. A detailed maternal-diet score may be collapsed into a binary adequate-versus-inadequate indicator, diluting or obscuring true associations. The article’s simulations quantify the damage. A Bland-Altman comparison of site-specific and harmonized measurements showed a mean difference of only −0.02, indicating minimal average bias, but 95% limits of agreement spanning −4.27 to 4.24, revealing substantial individual-level variability. In an 800-person simulation with a true association of about 0.70, regression on the harmonized measure attenuated the slope to near 0.10, making a meaningful effect appear nearly absent. A further simulation showed attenuation growing progressively more severe as measurement error increased. The authors recommend treating harmonization as measurement modeling, using tools such as regression calibration, simulation-extrapolation, latent-variable and multiple-indicator models, and documenting measurement error as structured uncertainty rather than technical noise. For some questions, they add bluntly, large datasets simply cannot provide a valid answer, and further tortuous analysis may be research waste.

The fourth threat is causal identification in the presence of rich covariates. Large datasets invite analysts to adjust for everything available, often with automated selection techniques such as stepwise regression, LASSO or high-dimensional propensity scores. But as covariates multiply, so does the risk of conditioning on mediators or colliders that distort the causal pathway. Adjusting for a mediator, such as inflammation biomarkers between pollution exposure and asthma, removes part of the true effect, while adjusting for a collider, such as a clinic visit rate affected by both exposure and unmeasured health status, opens spurious associations. Causal identification depends not on the sheer volume of data but on the underlying causal structure, which large-scale data alone may have little or no ability to reveal. Leaving out even a single major confounder produces residual bias that no sample size can correct.

Simulations again illustrate the trap. Progressively expanding an adjustment set beyond one key confounder yielded little additional improvement in the point estimate, which remained biased above the true effect even after five covariates, while confidence intervals widened substantially. And under exposure misclassification with sensitivity of 0.60 and specificity of 0.70, increasing sample size merely narrowed confidence intervals around systematically attenuated estimates without correcting the bias. The authors’ twelve-point guidance responds to all four threats: decide where the work sits on a continuum from exploration to targeted causation; evaluate statistical significance alongside effect sizes and minimally important differences; model site heterogeneity rather than ignoring it; treat harmonization as measurement modeling; use causal diagrams to guide adjustment whenever feasible; check covariate balance with standardized mean differences, not just model fit; embed sensitivity analyses, including vibration-of-effect and specification-curve approaches, as standard practice; report limitations structurally; consider falsification calibration against well-documented gold-standard results; consider multi-analyst approaches in which independent teams tackle the same pre-agreed question; enhance sharing, transparency and documentation; and favor encompassing analyses over fragmentation into least publishable units. The unifying message is that precision without structure does not guarantee truth. Valid inference in the big-data era, the authors conclude, depends less on the quantity of data than on the quality of design, measurement and careful causal reasoning, and on shifting the focus from statistical significance to validity, interpretability and robustness, so that biomedical research can realize the promise of large data without being precisely wrong.

Subject of Research: Statistical validity threats in large biomedical dataset research and practical guidance for addressing them

Article Title: Addressing threats to validity in large dataset research

Article References: Yoo, W., & Ioannidis, J. P. A. (2026). Addressing threats to validity in large dataset research. PLOS Medicine, 23(11), e1005282. https://doi.org/10.1371/journal.pmed.1005282

Image Credits: AI Generated

DOI: 10.1371/journal.pmed.1005282

Keywords: big data, statistical validity, p-value inflation, confounding, data harmonization, measurement error, causal inference, directed acyclic graphs, electronic health records, biomedical research, sensitivity analysis, reproducibility