Texture-Guided AI Sharpens Prostate Cancer Grading in Whole Slide Images

Prostate cancer remains one of the most frequently diagnosed malignancies in men, and the way pathologists grade it can determine whether a patient undergoes radical surgery, radiation, or active surveillance. A new deep learning architecture called TexTNet, described in Frontiers in Medicine, aims to make that grading more objective by teaching an artificial intelligence system to pay attention to the same subtle tissue textures that expert pathologists scrutinize under the microscope. The work addresses a long-standing problem in digital pathology: disagreement among observers, which studies report can reach 30 to 50 percent when general pathologists assign Gleason grades.

The Gleason grading system, the most powerful predictor of prostate cancer prognosis, classifies tumors by the architectural patterns of glandular tissue. Well-formed, distinct glands correspond to Grade 3, while fused, cribriform, or poorly formed structures indicate Grades 4 and 5. The critical boundary between Grade 3 and Grade 4 often hinges on subtle disruptions in glandular borders, precisely the kind of high-frequency visual detail that modern neural networks tend to smooth over. The authors of the new study argue that this is where existing hybrid models fall short.

Over the past decade, automated Gleason grading has evolved through several generations of techniques. Early computer-aided systems relied on handcrafted descriptors such as Local Binary Patterns (LBP) and Haralick textures, which captured chromatin density and stromal characteristics but could not model high-level semantic context. Convolutional neural networks, particularly U-Net variants, then enabled end-to-end feature learning and dramatically improved accuracy, with landmark systems achieving quadratic weighted kappa scores of 0.918 on biopsy test sets and near-perfect detection of benign versus malignant tissue. Yet convolutional networks are limited by the locality of their operations: their receptive fields are often too small to distinguish a cluster of tightly packed individual glands from a single large fused gland, the very distinction that separates Grade 3 from Grade 4.

Vision Transformers emerged as the answer to that limitation. By using self-attention to model dependencies across an entire image, architectures such as the Swin Transformer can capture the global structural coherence that convolutions miss. But the researchers behind TexTNet point out a fundamental weakness: Transformers tend to act as low-pass filters, prioritizing global shape over local texture. In histopathology, where patterns are defined by high-frequency textures and jagged boundary conditions rather than clean object silhouettes, this produces over-smoothed segmentation masks that wash out the morphological cues pathologists depend on.

TexTNet’s central innovation is to make texture a guiding force rather than an afterthought. Instead of simply concatenating or summing features from a convolutional branch and a Transformer branch, as most hybrid models do, the architecture feeds LBP texture features directly into the query projection of the Swin Transformer. This creates what the authors call a Texture-Guided Attention mechanism: the attention scores become sensitive to both spatial position and local morphological nuance, directing the model toward texture-rich regions such as glandular borders and nuclear clusters. Because LBP relies on relative pixel inequalities rather than absolute intensities, the resulting features are also inherently robust to the stain variations that plague multi-site pathology data.

The architecture is supported by several purpose-built modules. A Dual Path Squeeze-Fusion-Excitation (DP-SFE) module acts as a co-attention bridge between the texture stream and the semantic stream, dynamically recalibrating feature responses in both channel and spatial dimensions to mitigate the boundary blurring introduced by the Transformer’s patch-merging process. Asymmetric Convolution Blocks (ACB) reinforce horizontal and vertical structural edges in the skip connections, preserving fine glandular boundaries that would otherwise degrade during downsampling. In the decoder, a Cross-Directional Feature Aggregator (CDFA) reconstructs precise positional information along orthogonal directions and suppresses stromal noise, while a Refined Segmentation Head aggregates multi-scale features from all decoder stages into the final pixel-level map.

The performance gains reported are substantial. On the SICAPv2 dataset, which provides pixel-level annotations for Gleason patterns 3, 4, and 5, TexTNet achieved a mean Dice Similarity Coefficient of 90.59 percent, with per-grade scores of 90.67, 90.88, and 90.22 percent respectively, outperforming established baselines including TransUNet, MFTrans, DHUnet, CTNet, and MaskHIT. On the large-scale PANDA challenge dataset, TexTNet reached a mean Dice of 88.07 percent, again leading all compared methods. Ablation studies showed that naive fusion of LBP features yielded only marginal gains, while the full texture-guided attention mechanism lifted mean Dice from 86.18 to 89.21 percent, and adding DP-SFE pushed it to 90.59 percent, confirming that the way texture is integrated matters as much as the texture itself.

Perhaps most striking is the model’s generalization beyond its training domain. On the HuBMAP dataset of PAS-stained prostate tissue, where the task is to segment functional tissue units in healthy specimens, TexTNet achieved 97.40 percent accuracy and a Dice of 86.93 percent, demonstrating cross-stain adaptability. On ProstateX, an MRI dataset from patients with elevated PSA levels, the same architecture outperformed prior methods in segmenting the transition zone, peripheral zone, and urethra, with a Dice of 97.21 percent in the transition zone. The authors attribute this cross-modality transfer to the stain-invariant nature of LBP descriptors and the DP-SFE module’s role as an implicit domain adapter that recalibrates channels to isolate robust structural features.

The researchers are candid about the system’s limitations. Transformer-based architectures carry significant computational overhead, which hinders real-time deployment in resource-constrained clinical settings without specialized hardware. The ablation tables show the full model requires nearly 30 million parameters and 16.8 GFLOPS, with inference times rising from about 11 milliseconds for the baseline to roughly 25 milliseconds. The team also documented failure cases in which atypical stromal morphology visually mimics malignant glands, disrupting the LBP descriptors and producing fragmented boundaries, and they propose uncertainty-aware learning to flag such regions for human review.

Looking forward, the authors plan to extend TexTNet to multimodal datasets combining histopathology with radiology, explore self-supervised learning to reduce dependence on expensive pixel-level annotations, and develop lightweight versions for real-time inference on gigapixel whole slide images. If those engineering challenges can be solved, texture-guided attention of this kind could move from benchmark leaderboards into the diagnostic workflow itself, offering pathologists a quantitative second opinion precisely where inter-observer variability is highest: the subtle, texture-defined boundary between indolent and aggressive disease.

Subject of Research: Texture-guided deep learning architecture for prostate cancer Gleason grade segmentation in whole slide images

Article Title: TexTNet: a texture-guided hierarchical architecture for prostate cancer Gleason segmentation in whole slide images

Article References: Arshad, M., Chengliang, W., Wajeeh Us Sima, M., Shaikh, J. A., Alkhalaf, S., & Alturise, F. (2026). TexTNet: a texture-guided hierarchical architecture for prostate cancer Gleason segmentation in whole slide images. Frontiers in Medicine, 13, Article 1815031. https://doi.org/10.3389/fmed.2026.1815031

Image Credits: AI Generated

DOI: 10.3389/fmed.2026.1815031

Keywords: prostate cancer, Gleason grading, whole slide images, deep learning, Swin Transformer, texture-guided attention, local binary patterns, image segmentation, computational pathology, SICAPv2, PANDA dataset, medical imaging AI