Computational researchers Mia Venter and Kevin J. Naidoo have demonstrated that unsupervised machine learning applied to aberrant glycogene expression profiles can stratify early-stage breast cancer into seven distinct molecular subclasses. Reporting in Discover Chemistry, the team investigated how altered cellular glycosylation—an enzymatic mechanism wherein carbohydrate moieties are conjugated to proteins and lipids across the endoplasmic reticulum and Golgi complex—reflects fundamental tumor heterogeneity. In malignant tissues, this non-template-driven biosynthetic pathway becomes profoundly dysregulated, altering cell-surface glycoconjugates that act as immune and signaling receptors. By analyzing gene expression profiles from 785 female stage I and stage II breast cancer samples cataloged in The Cancer Genome Atlas, the investigators demonstrated that glycosylation-specific transcriptional programs uncover clinically relevant biological groupings that standard gene expression profiling frameworks can overlook.
Traditional subtyping diagnostics utilized in clinical oncology, such as the PAM50 and MammaPrint genomic assays, rely on broader gene expression profiles to guide clinical understanding and patient care. While these standard tools distinguish general molecular classes, they frequently leave unexplained the precise biochemical mechanisms that drive tumor progression or dictate therapeutic responsiveness in individual patients. To address this biological gap, the research team focused on glycogenes, the specific genes responsible for encoding enzymes that catalyze glycosylation reactions. Because the spatial and temporal distribution and absolute abundance of glycogenes determine the architecture of cellular glycoconjugates, the researchers hypothesized that quantifying aberrant glycogene expression would expose underlying regulatory networks. Mining primary untreated RNA-sequencing data, the authors curated a focused panel of 341 glycogenes demonstrating standard deviations greater than 0.5 across early-stage patient samples.
To overcome the curse of dimensionality inherent to genomic datasets—wherein candidate genes dramatically outnumber available biological samples—the investigators developed and implemented the Growing Hierarchical Self-Organising Representation Map, or GHSORM. This specialized unsupervised machine learning framework pairs a Denoising Autoencoder with a series of hierarchically arranged Self-Organising Maps. At the input stage, the Denoising Autoencoder corrupts normalized expression values using Gaussian noise with a standard deviation of 0.2, training hidden artificial neurons through hyperbolic tangent activation functions to preserve fundamental topological relationships while eliminating observational noise. The encoded representation layer is then projected onto self-organizing lattices that expand dynamically according to user-defined quantization error thresholds, permitting deeper hierarchical layers to resolve fine-grained biological clusters beyond the capacity of static k-means or standard hierarchical clustering algorithms.
Applying GHSORM to the early-stage breast cancer cohort separated the patient samples into seven distinct Glycogene Expression Clusters, designated GEC1 through GEC7. The foundational layer of the model achieved immediate separation between Luminal and Basal tumors, while the secondary SOM layers uncovered granular subdivisions within these classical categories. Through a brute-force grid search aimed at minimizing intra-cluster similarity relative to inter-cluster distance, the optimal Pre-Processor GHSORM architecture achieved a Davies-Bouldin Index of 3.17, outperforming conventional k-means and hierarchical clustering implementations which yielded inferior indices of 3.26 and 3.50, respectively. Cross-referencing these groups with standard PAM50 classifications revealed a Normalized Mutual Information metric of 0.52, confirming substantial biological overlap while highlighting key divergences where Luminal A and Luminal B samples dispersed across multiple distinct glycogene clusters.
To identify the minimal genetic footprint capable of distinguishing these seven subclasses with high diagnostic precision, the authors engineered an in-house feature selection framework termed Recursive Feature Elimination with Strikeout, or RFES. Wrapped around a Random Forest classification architecture, RFES iteratively prunes non-informative genes based on their Mean Decrease in Accuracy across clusters. Crucially, the algorithm incorporates a strikeout parameter that allows classification accuracy to temporarily fluctuate, preventing premature optimization and facilitating an exhaustive exploration of the 341-feature space. Applied alongside class-balancing techniques—random undersampling for dominant classes and synthetic inflation via RACOG for minority groups—RFES isolated a refined signature of 60 glycogenes capable of classifying samples into their respective GECs with 90.48 percent accuracy across a five-fold cross-validation testing partition.
Detailed interrogation of the 60-glycogene signature revealed that unique biochemical pathways define each individual cluster. For example, in GEC6, which predominantly encompasses HER2-enriched tumors, key classification features included sulfatases ARSB and ARSG alongside the carbohydrate sulfotransferase CHST11. These enzymes directly modulate the sulfation states of glycosaminoglycans such as chondroitin sulfate and dermatan sulfate within the extracellular matrix. Sulfated glycosaminoglycans establish physical, size-selective microenvironmental barriers around malignant cells while simultaneously functioning as sensitive co-receptors that initiate downstream oncogenic signaling cascades. By contrast, GEC4 and GEC5 divided Basal-like breast cancers into two biologically discrete subgroups, each characterized by mutually exclusive glycogene expression profiles that correlate with distinct tumor microenvironment compositions.
To clarify how glycosylation patterns intersect with tumor-infiltrating immune cells, the authors applied Cibersort deconvolution to 773 tumor samples, mapping relative leukocyte distributions across clusters. The analysis revealed marked immunological divergence; despite sharing similar baseline PAM50 assignments, GEC2 displayed significantly elevated levels of monocytes and CD8 positive T-cells compared to GEC3. Furthermore, GEC5 exhibited substantially higher abundances of CD8 positive cytotoxic T-cells and regulatory T-cells compared to GEC4, which instead demonstrated an enrichment of resting memory CD4 positive T-cells. Correlating single-sample gene set enrichment scores against leukocyte distributions confirmed that macrophage polarization tracked glycosaminoglycan synthesis: biosynthesis of dermatan, chondroitin, and heparan sulfate correlated positively with pro-tumorigenic M2 macrophages and negatively with tumoricidal M1 macrophages across GEC5, GEC6, and GEC7.
The researchers subsequently tested the clinical utility of this subtyping schema by projecting pre-treatment expression profiles from 984 patients enrolled in the adaptive I-SPY2 clinical trial into the GEC framework using their 60-glycogene RFES model. Historically, grouping patients retroactively by Response Predictive Subtypes, or RPS-5, achieved a 7 percent improvement in pathologic complete response compared to conventional hormone receptor subtyping. When the authors evaluated patient distribution and treatment regimens under the GEC classification, modeled counterfactual treatment assignments yielded an estimated overall pathologic complete response rate of 86 percent, representing a 28 percent improvement over RPS-5 and a 35 percent improvement over standard receptor subtyping.
Examination of therapeutic arms within specific GECs supported the mechanistic relevance of glycogene-based stratification. Within GEC5, which models an immunotherapy-responsive Basal subtype, all nine patients who received neoadjuvant paclitaxel combined with the immune checkpoint inhibitor pembrolizumab attained complete pathologic responses. Gene regulatory network analysis utilizing Arboreto and ClusterProfiler confirmed that upregulated glycogenes in GEC5 directly predict the expression of non-glycogenes governing B-cell and T-cell activation. Conversely, in GEC6, four of five patients treated with paclitaxel, the angiopoietin antagonist AMG 386, and trastuzumab achieved complete responses. Regulatory network modeling demonstrated that GEC6 glycogenes predict non-glycogene networks regulating angiogenesis, extracellular matrix organization, and the downstream ERK1 and ERK2 signaling cascade.
Despite these encouraging modeling outcomes, the authors emphasize essential methodological limitations that restrict immediate clinical application. The 90.48 percent classification accuracy of the 60-glycogene panel remains below the stringent reliability thresholds demanded for frontline clinical diagnostics. Furthermore, because the transcriptional data were derived from bulk tumor sequencing, the discrete cellular lineages contributing to observed glycogene alterations cannot be definitively resolved without single-cell validation. The authors also note that transcript abundance does not automatically guarantee equivalent glycomic alterations, as post-translational mechanisms strongly govern final glycan assembly. Consequently, while GHSORM demonstrates that glycogene expression reliably groups tumors according to mechanistic drug vulnerabilities, prospective experimental validation linking specific glycan structures to oncogenic pathways will be essential to translate these computational discoveries into verified clinical interventions.
Subject of Research: Machine learning subtyping of breast cancer from glycogene expression
Article Title: Unsupervised machine learning identifies early stage breast cancer subclasses from aberrant glycogene expression and informs personalised drug treatment
Article References: Venter, M., & Naidoo, K. J. (2026). Unsupervised machine learning identifies early stage breast cancer subclasses from aberrant glycogene expression and informs personalised drug treatment. Discover Chemistry, 3(1), Article 630. https://doi.org/10.1007/s44371-026-01070-0
Image Credits: AI Generated
DOI: 10.1007/s44371-026-01070-0
Keywords: glycogene expression, breast cancer subtyping, GHSORM algorithm, Recursive Feature Elimination with Strikeout, unsupervised machine learning, Denoising Autoencoder, Self-Organising Maps, tumor microenvironment, glycosaminoglycan sulfation, Cibersort deconvolution, pathologic complete response, I-SPY2 trial, pembrolizumab response, angiogenesis regulation, macrophage polarization, computational oncology, The Cancer Genome Atlas
News Source: Nathaniel Bowman. (November 14, 2026). Machine Learning Identifies Glycogene Patterns That Define Early Breast Cancer Subtypes. Scienmag.
