Statistical methods that quietly assume independence between data points often stumble on real-world biology, where correlations are everywhere. Now a team at the Icahn School of Medicine at Mount Sinai has rebuilt one of the workhorses of modern statistics, the whitening transformation, on a probabilistic foundation that makes it dramatically faster and more accurate on the enormous datasets that define contemporary genetics. The study, published in PLOS Genetics, introduces a regularized whitening approach that avoids ever explicitly estimating the covariance matrix it depends on, cutting computation from cubic to linear time in the number of features.
Whitening transformations are a family of linear algebra operations that remove correlation structure from data. For a data matrix with n samples and p features, the procedure estimates the covariance between features and multiplies the data by a transformation matrix so that the result has an identity covariance matrix. The most intuitive variant, known as zero-phase components analysis or ZCA whitening, rotates the data along its principal axes, scales them, and rotates them back to the original axes. Although the transformation is not uniquely defined, since different rotations all yield identity covariance, this version minimizes the total squared distance between the original and transformed data.
The trouble is that the true covariance matrix is never known and must be estimated from the data. An exact whitening transformation is only possible when the number of samples exceeds the number of features, so that the sample covariance matrix is invertible, and even then the transform can become numerically unstable when trailing singular values are very small. In high-dimensional settings where features outnumber samples, the sample covariance matrix is no longer full rank, some singular values are exactly zero, and some form of regularization becomes unavoidable. Computationally, evaluating the covariance and whitening matrices becomes infeasible for large numbers of features, with the standard algorithm becoming very expensive beyond a few thousand features.
The Mount Sinai group, led by Gabriel Hoffman with Christian Dillard, Kiran Girdhar and Panos Roussos, reframed the problem probabilistically. They model the observed data as a matrix normal distribution with an inverse-Wishart prior placed on the covariance between columns. Because the inverse-Wishart is the conjugate prior, the posterior expectation of the covariance is a convex combination of the sample covariance and a prior target matrix, which the authors set to a scaled identity matrix using the average variance across features. The result is a single shrinkage hyperparameter, estimated from the data itself through an empirical Bayes approach the authors call the Gaussian Inverse Wishart empirical Bayes, or GIW-EB, model. Integrating out the covariance matrix turns the marginal distribution into a multivariate Student-t, and the data enters the likelihood only through the sample singular values.
This formulation yields a striking mathematical simplification. The whitening transformation retains the original singular vectors of the data and acts only by modifying the singular values, shrinking them toward a floor so that the transformed covariance is always positive definite and invertible even when the sample covariance is not. From this spectral view, the authors derived a general identity showing that only the singular vectors corresponding to nonzero singular values are needed to apply the transform. Combined with a rank-k partial singular value decomposition, the algorithm’s complexity becomes linear in both the number of samples and the number of features, a dramatic improvement over the naive method that is cubic in the number of features. In simulations with 5,000 samples and an increasing number of features, the GIW-EB method ran in under two minutes and a low-rank version with k equal to 50 ran in under one minute, while the standard method became intractable in both time and memory.
Accuracy was tested out of sample, the setting that matters most in genetics. The team simulated data from multivariate Gaussian distributions with 500 samples and between 20 and 2,000 features under four correlation scenarios: constant correlation, autocorrelation, four blocks of constant correlation, and four blocks of autocorrelation. A whitening transformation estimated from one subset of the data was applied to fresh samples, and error was measured as the root mean squared deviation of the transformed covariance from the identity matrix. Across all scenarios, GIW-EB consistently gave the lowest out-of-sample error, with the largest gains when the correlation matrix was low rank and the eigen-spectrum was dominated by the first few eigen-values. A low-rank variant was most competitive when the true covariance was itself low rank.
Because many analyses of genome-wide association study summary statistics apply whitening transformations estimated from an external reference panel, the authors benchmarked the method on real genetic data from the 1000 Genomes Project. Individuals of European, Asian and African ancestry were analyzed separately over independent linkage disequilibrium blocks, with the transformation trained on half the samples and evaluated on the other half. Despite genetic data being neither continuous nor multivariate Gaussian, shrinkage methods performed well across most windows in European individuals, and GIW-EB matched or beat competing methods. The implicit covariance algorithm gave a wall time of five minutes compared with three to thirty hours for other methods, and in African and Asian ancestry populations GIW-EB gave lower error than fixing the shrinkage parameter in 75.2 to 91.2 percent of genome windows, depending on the population.
The headline application was imputing z-statistics for genetic variants not directly genotyped in a genome-wide association study. Correlation between z-statistics of two variants mirrors the linkage disequilibrium between them, so unobserved statistics can be predicted from observed ones and the correlation structure of the region. Using summary statistics for 5.6 million variants from a schizophrenia GWAS of 58,000 cases and 77,000 controls of European ancestry, the team imputed z-statistics for 15 million variants with a reference panel of just 407 unrelated European-ancestry individuals from the 1000 Genomes Project, sliding a one megabase window with 250 kilobase flanks across all autosomes. Accuracy, checked against observed values for roughly 460,000 randomly selected variants, was highest for GIW-EB, which gave the smallest genome-wide root mean squared error and the smallest squared error across the range of minor allele frequencies. The implicit covariance approach completed the genome-wide analysis in under an hour, versus four to 126 hours for competing methods.
As a proof of principle, the authors turned to tandem repeats, stretches of DNA poorly captured by standard genotyping panels. Drawing on a population reference panel of 1.7 million tandem repeats built from 1000 Genomes Project donors, they integrated it with summary statistics from a study of genetic regulation of gene expression in the human brain using nearly 4,000 bulk RNA-seq samples. For three disease-relevant genes, the imputation identified tandem repeats explaining the genetic regulatory signal. At C9orf72, a key risk gene for amyotrophic lateral sclerosis where expansion of an intronic hexanucleotide repeat confers disease risk and affects gene expression, the imputation identified tandem repeats explaining the genetic regulatory signal, while at TSPAN14 and TYK2, both implicated in Alzheimer's disease risk, the analysis found tandem repeats with stronger associations to gene expression than the SNPs in the region.
Still, the practical payoff is considerable: whitening transformations underlie phenotype prediction, statistical fine-mapping, multiple testing correction, anomaly detection and knockoff simulation, usually implicitly. The new methods are implemented in the open-source R packages decorrelate and imputez, and the authors suggest that as genomic datasets keep expanding, the implicit covariance approach will extend whitening to settings such as single-cell gene expression and larger reference panels where existing methods are simply intractable.
Subject of Research: A probabilistic whitening transformation for removing correlation structure in ultra-high-dimensional genetic data
Article Title: Fast probabilistic whitening transformation for ultra-high dimensional genetic data
Article References: Hoffman, G. E., Dillard, C. P., Girdhar, K., & Roussos, P. (2026). Fast probabilistic whitening transformation for ultra-high dimensional genetic data. PLOS Genetics, 22(10), e1012320. https://doi.org/10.1371/journal.pgen.1012320
Image Credits: AI Generated
DOI: 10.1371/journal.pgen.1012320
Keywords: whitening transformation, linkage disequilibrium, empirical Bayes, covariance shrinkage, GWAS summary statistics, z-statistic imputation, schizophrenia, 1000 Genomes Project, tandem repeats, eQTL, computational statistics, PLOS Genetics
News Source: Juliet Wilcox. (October 28, 2026). New Whitening Method Speeds Up Giant Genetic Data Analysis. Scienmag.
