Machine Learning Outperforms Standard Linear Genomic Prediction Only Under Specific Genetic Architectures

A comprehensive simulation study published in Genetics Selection Evolution demonstrates that machine learning models outperform conventional linear prediction in genomic breeding only under specific genetic architectures. Evaluating livestock breeding populations, the authors compared three machine learning models, including random forest, support vector regression, and a multilayer perceptron with residual network connections, against genomic best linear unbiased prediction, commonly known as GBLUP. The investigation revealed that machine learning methods surpass GBLUP only when target traits are controlled by a small number of quantitative trait loci, specifically one hundred or fewer. Under polygenic architectures where thousands of loci govern variation, conventional linear modeling consistently matched or exceeded the performance of machine learning alternatives. Furthermore, the operational advantage among machine learning models depended directly on dataset scale: random forest outperformed GBLUP with small reference populations, whereas deep neural networks required large training populations exceeding ten thousand animals to demonstrate any measurable benefit over standard linear evaluation.

Genomic prediction represents an essential tool in contemporary animal and plant breeding programs, predicting genomic estimated breeding values of young selection candidates from dense molecular markers before phenotypes are expressed. Operational breeding evaluations currently rely on linear statistical frameworks, most notably GBLUP and Bayesian approaches. These classical models assume that complex traits are governed by the infinitesimal model, in which single nucleotide polymorphism effects follow a normal distribution with equal variance. In contrast, machine learning frameworks impose minimal distributional assumptions on marker effects, offering mathematical flexibility to capture complex non-linear relationships between genotypes and phenotypes. While earlier empirical breeding studies reported that machine learning algorithms yielded superior prediction accuracy, those investigations were largely restricted to small training datasets of fewer than five thousand individuals. Moreover, past empirical evaluations assessed prediction accuracy by correlating model outputs with observed phenotypes rather than true additive breeding values, inadvertently inflating machine learning performance by capturing non-additive genetic variation.

To clarify these unresolved questions without confounding factors from field records, the researchers performed computer simulations using QMSim to generate realistic livestock populations. The simulation mimicked a historical pig population to reproduce authentic linkage disequilibrium patterns observed in commercial swine. The historical cohort initiated with three thousand animals of equal sex ratio, contracted gradually to two hundred individuals by generation 1400, and remained constrained at that size until generation 1600 to establish linkage disequilibrium. Population size then expanded linearly to 10,200 individuals by generation 1605. The simulated genome comprised ten pairs of autosomes, each measuring one hundred centimorgans, containing 80,000 segregating loci distributed randomly across chromosomes. Following the historical phase, sixteen consecutive generations were simulated with an equal sex ratio and litter size of two. In each generation, the two hundred males with highest estimated breeding values for a selection trait were mated to ten thousand females, mirroring practical selection schemes while preserving realistic genomic structure.

Because artificial selection inherently reduces genetic variance and heritability across generations, the investigators simulated an additional unselected target trait across generations eleven through sixteen to compare prediction accuracy objectively across scenarios. The final dataset contained complete genotype and phenotype records for 120,000 animals across these six generations, showing a characteristic U-shaped allele frequency distribution. For genomic prediction, the authors selected 20,000 segregating loci that were not sampled as causal variants, establishing a uniform marker allele frequency distribution and linkage disequilibrium matching commercial swine populations. The study evaluated twenty-four distinct scenarios comprising combinations of three trait architectures, two heritability levels, and four reference population sizes. Trait architectures contained 50, 100, or 5000 quantitative trait loci sampled from the genome using a nested design where smaller sets were subsets of larger sets. Additive locus effects were sampled from a standard normal distribution, and phenotypes were generated with target heritabilities of 0.1 and 0.5.

Reference populations contained 1000, 2000, 10,000, or 50,000 animals drawn from generations eleven through fifteen in nested subsets, while five hundred generation sixteen individuals formed the test cohort. GBLUP was implemented using genomic relationship matrices computed via calc_grm, with variance components and breeding values estimated through MTG2. Random forest was built using Scikit-learn, with tree numbers fixed at 1500 after evaluating computational trade-offs, alongside grid search over maximum features, maximum depth, and leaf sizes. Support vector regression mapped marker genotypes into higher-dimensional feature space, evaluating linear, radial basis function, and polynomial kernels. Deep learning was implemented with MLP-ResNet in Keras. Initial experiments with standard feedforward neural networks encountered severe output collapse, generating identical predictions across validation animals regardless of genotype; introducing residual shortcut connections successfully resolved this failure. Hyperparameters for all machine learning models were tuned independently for each scenario using one reserved simulation replicate divided between training and validation cohorts.

Across twenty analyzed simulation replicates per scenario, the number of quantitative trait loci emerged as the defining factor governing prediction accuracy for random forest and MLP-ResNet, whereas GBLUP and support vector regression demonstrated minimal sensitivity to locus numbers. Random forest prediction accuracy dropped dramatically as the number of loci increased, showing an average decline of 0.13 between traits with 50 loci and 5000 loci. The authors attributed this degradation to feature partitioning through binary decision splits. When a trait is controlled by few loci, each variant explains substantial genetic variance, increasing the probability that algorithms select highly informative markers at initial splits that define tree structure. As locus counts multiply, individual effects dilute, causing early splits to incorporate non-informative noise markers and diminishing prediction accuracy. For MLP-ResNet, prediction accuracy decreased by an average of 0.06 between 50 and 5000 loci in large reference populations, because gradient-based optimization struggles when predictive signals disperse across thousands of minor loci.

Direct model comparisons demonstrated that machine learning algorithms achieved higher prediction accuracy than GBLUP only under restricted circumstances. Random forest outperformed GBLUP exclusively when traits were influenced by 50 loci and reference populations were small, particularly at 1000 or 2000 individuals. This advantage was more prominent under low heritability of 0.1 than high heritability of 0.5. However, as reference populations expanded to 10,000 or 50,000 animals, or when traits were governed by 100 or 5000 loci, GBLUP consistently surpassed random forest. In contrast, MLP-ResNet achieved no advantage in small training datasets, but outperformed GBLUP when reference population sizes reached at least 10,000 animals combined with small trait architectures of 50 or 100 loci. Support vector regression failed to exceed GBLUP in any scenario, exhibiting marginally lower accuracy across all combinations. The researchers noted that past empirical studies reporting superior accuracy for support vector regression likely reflected non-additive genetic variation present in phenotypes rather than true additive breeding value prediction.

Beyond accuracy correlations, the authors evaluated dispersion bias using the regression slope of true breeding values on predicted values, where a coefficient of one represents unbiased prediction. GBLUP exhibited the lowest dispersion bias across all scenarios, maintaining regression coefficients close to one, particularly when reference populations exceeded 2000 individuals. Random forest displayed under-dispersion across all evaluated conditions. MLP-ResNet exhibited substantial dispersion instability when applied to small reference populations under low heritability, generating extreme regression coefficients exceeding ten, though bias decreased substantially and regression slopes approached one in larger reference populations under high heritability. Computational timing benchmarks conducted on single-processor hardware with 50,000 reference individuals revealed pronounced differences among algorithms. MLP-ResNet was the fastest model, requiring 16 to 80 minutes to compute, followed by random forest at 82 to 520 minutes. GBLUP required 299 to 645 minutes including variance component estimation, while support vector regression required 584 to 6572 minutes.

Supplementary analyses evaluating an extreme heritability setting of 0.01 supported the primary conclusions of the study. Under very low heritability, random forest maintained an advantage over GBLUP when traits possessed 50 loci even with reference populations larger than 10,000, confirming that the relative benefit of random forest expands as heritability decreases. However, MLP-ResNet displayed substantial variability across replicates, showing vulnerability to suboptimal hyperparameter configurations. The authors observed that while most quantitative traits are polygenic, certain traits follow oligogenic-polygenic mixed architectures where major genes explain significant variance proportions. Examples include grain dimensions and thousand-grain weight in rice, where five major loci explain over thirty percent of phenotypic variation, canine body size where six genes account for nearly half of variation, dairy cattle milk fat percentage, and pig litter size. For such oligogenic traits, random forest provides a promising model in small reference cohorts, whereas MLP-ResNet offers computational efficiency and high accuracy in large commercial datasets.

The researchers noted important limitations in their investigation, including the restriction to purely additive genetic effects without dominance or epistasis, and the assumption of normally distributed locus effects rather than heavy-tailed distributions such as gamma distributions. Furthermore, hyperparameter tuning presents practical operational challenges for commercial breeding implementation, as optimal parameter combinations varied across quantitative trait loci counts, heritabilities, and population sizes. Unlike linear models that operate reliably without extensive hyperparameter optimization, machine learning models required scenario-specific tuning to maintain predictive validity. The authors concluded that before implementing machine learning models or neural networks in genomic prediction or polygenic risk score calculations, researchers must rigorously evaluate them against simulated datasets across varying numbers of loci and reference population sizes. These findings demonstrate that machine learning models provide genuine predictive advantages over GBLUP only when traits deviate from infinitesimal assumptions, establishing clear empirical boundaries for machine learning applications in modern agricultural genetics.

Subject of Research: Comparative performance of machine learning models and GBLUP in genomic prediction

Article Title: Impact of QTL number, heritability and reference population size on the benefit of machine learning models over GBLUP in genomic prediction

Article References: Yang, J., Wientjes, Y. C. J., Calus, M. P. L., & Duenk, P. (2026). Impact of QTL number, heritability and reference population size on the benefit of machine learning models over GBLUP in genomic prediction. Genetics Selection Evolution, 58(1), Article 49. https://doi.org/10.1186/s12711-026-01065-6

Image Credits: AI Generated

DOI: 10.1186/s12711-026-01065-6

Keywords: genomic prediction, machine learning, GBLUP, random forest, multilayer perceptron, support vector regression, quantitative trait loci, animal breeding, Impact, number, heritability, reference