Agricultural engineers and artificial intelligence researchers have established an explainable machine learning methodology that uses visual decision heatmaps to design hybrid convolutional and vision transformer architectures for grapevine semantic segmentation. In modern precision viticulture, automated vision systems deployed on agricultural machinery and autonomous robots require pixel-level identification of plant structures to facilitate yield estimation, biomass assessment, and targeted canopy management. However, distinguishing complex agricultural components such as leaves, grape clusters, and woody vine trunks remains difficult because of irregular plant morphology, variable canopy structures, fluctuating natural illumination, and visual clutter across commercial fields. To address whether distinct deep learning paradigms can be effectively united rather than combined arbitrarily, investigators evaluated baseline models using gradient-weighted class activation mapping to quantify the spatial evidence guiding their predictions before engineering and benchmarking dual-encoder hybrid networks.
Semantic segmentation operates by assigning a categorical semantic label to every individual pixel within an agricultural image. Convolutional neural networks, including DeepLabV3+ with an EfficientNet-B4 backbone, process visual input through hierarchical receptive fields and parallel dilated convolutions that excel at extracting localized spatial textures, fine-grained geometric edges, and object boundaries. In contrast, vision transformers, exemplified by SegFormer equipped with a hierarchical Mix Vision Transformer backbone, leverage multi-head self-attention mechanisms to establish long-range contextual relationships across distant regions of an image. Although the conceptual complementary nature of local feature extraction and global context modeling has inspired hybrid models in medical imaging, researchers have lacked systematic methods to determine how these separate network families allocate visual attention when analyzing real-world viticultural environments.
To investigate the internal decision-making processes of these deep learning architectures, the research team analyzed model behaviors across two independent field datasets collected in commercial Italian vineyards. The first dataset was acquired in the Conte Spagnoletti Zeuli vineyard in Andria, cultivating the Vitis vinifera cultivar Nero di Troia, utilizing a Microsoft Azure Kinect camera mounted horizontally on a tripod positioned at distances between 1.5 and 2.0 meters from the vine row. This setup framed multiple plants at a resolution of 2048 by 1536 pixels under challenging conditions characterized by dense foliage, intense shadows, and strong natural backlighting. The second dataset was collected in San Donaci using an Intel RealSense D435 camera mounted in portrait orientation on a mobile robot traveling between 0.8 and 1.0 meters from rows of the Negroamaro grape cultivar, recording single-vine frames at 1280 by 720 pixels across phenological stages extending from active vegetative growth through grape maturation.
Rather than treating explainable artificial intelligence as a post-hoc diagnostic tool intended solely to visualize model failures after deployment, the researchers incorporated Gradient-weighted Class Activation Mapping directly into the network engineering workflow. By backpropagating class scores obtained from predicted segmentation masks to functionally equivalent decoder layers situated immediately prior to the final segmentation heads, the researchers produced comparative spatial heatmaps for target classes. They evaluated these heatmaps using objective localization metrics, including the Pointing Game, which measures whether the maximal activation peak falls within ground-truth object annotations; the Energy-Based Pointing Game, which computes the proportion of total activation energy concentrated within object boundaries; and IoU-CAM, which evaluates spatial intersection over union against binarized attribution masks.
The quantitative explainability analysis revealed that while DeepLabV3+ and SegFormer achieved broadly comparable baseline segmentation performance, their underlying visual reasoning diverged significantly across semantic categories. For the grape cluster class, the Energy-Based Pointing Game revealed statistically significant differences, with DeepLabV3+ concentrating between 78.4 percent and 81.9 percent of its total attribution energy directly within grape boundaries, compared to 72.9 percent to 73.9 percent for SegFormer across the Zeuli and San Donaci datasets. SegFormer consistently distributed its explanatory activation across adjacent leaves and background foliage, indicating an expansive contextual approach, whereas the convolutional network remained strictly localized. For leaf classification, DeepLabV3+ achieved significantly higher Pointing Game scores of 100.0 percent and 93.7 percent on the respective datasets compared to 59.3 percent and 81.0 percent for SegFormer, demonstrating that the convolutional architecture anchored its peak confidence on discrete local structures while the transformer integrated broader canopy regions.
Guided by these concrete empirical observations rather than theoretical assumptions, the authors formulated an explicit architectural hypothesis: integrating the concentrated local feature representations of convolutional backbones with the broad contextual representations of vision transformers within a unified dual-encoder network could resolve class-specific ambiguities. To test this hypothesis, the researchers engineered four distinct hybrid configurations pairing an EfficientNet-B4 convolutional encoder and a MiT-B2 transformer encoder. Because EfficientNet-B4 generates high-resolution geometric feature maps at its initial stage that have no dimensional counterpart in the transformer hierarchy, this early convolutional tensor was routed directly to the network decoder. Across subsequent hierarchical feature levels, the parallel representations were fused using two alternative mechanisms: direct channel concatenation followed by channel-compressing convolutions, or a linear cross-attention module allowing convolutional query projections to selectively attend to transformer key and value embeddings.
The fused multi-scale feature pyramids were subsequently coupled with two distinct decoding heads to assess the role of spatial aggregation. One decoder employed a lightweight multi-layer perceptron architecture adapted from SegFormer that upsamples and concatenates multi-scale features to a uniform spatial resolution. The second decoder implemented the atrous spatial pyramid pooling mechanism from DeepLabV3+, which applies parallel dilated convolutions at multiple sample rates to the deepest feature map before merging the result with high-resolution low-level features through separable convolutions. Across these structural pairings, Hybrid 1 combined cross-attention with atrous spatial pyramid pooling, Hybrid 2 combined cross-attention with the multi-layer perceptron head, Hybrid 3 paired feature concatenation with atrous pooling, and Hybrid 4 paired feature concatenation with the lightweight multi-layer perceptron decoder.
Experimental evaluation over five-fold cross-validation demonstrated that performance improvements in hybrid models depend heavily on the interaction between the feature integration mechanism, decoder complexity, and dataset environment. Hybrid 4 emerged as the only architecture that consistently outperformed the best baseline models across all target classes in both vineyard environments. On the San Donaci dataset, Hybrid 4 achieved intersection over union scores of 71.31 percent for grapes, 94.10 percent for leaves, and 65.10 percent for vine trunks, exceeding the baseline convolutional model scores of 68.44 percent, 93.72 percent, and 62.63 percent, as well as baseline transformer scores of 67.72 percent, 93.69 percent, and 63.16 percent. In the more complex Zeuli dataset, Hybrid 4 maintained superior performance, attaining 56.01 percent intersection over union for grapes, 83.11 percent for leaves, and 67.84 percent for trunks, compared to baseline convolutional scores of 55.41 percent, 82.73 percent, and 66.79 percent.
The experimental outcomes also illustrated that increased architectural complexity does not guarantee superior robustness across varying field conditions. Although Hybrid 1 achieved the highest individual segmentation scores on the San Donaci dataset with 71.46 percent intersection over union for grapes and 66.20 percent for trunks, its performance deteriorated sharply on the Zeuli dataset, dropping to 50.67 percent for grapes and 62.74 percent for trunks, falling below both single-encoder baselines. This performance drop highlights that cross-attention mechanisms can become sensitive to environmental visual degradation, such as direct camera backlighting, canopy shadows, and varying foliage density, whereas direct channel concatenation within Hybrid 4 provided a more stable and resilient synthesis of dual-encoder representations across distinct grape cultivars and capture configurations.
Computational efficiency assessments conducted on a desktop workstation featuring an NVIDIA GeForce RTX 4090 GPU revealed distinct latency profiles among the evaluated networks. On the San Donaci dataset, baseline SegFormer recorded the lowest per-image inference time at 0.0737 seconds, while baseline DeepLabV3+ required 0.1431 seconds due to dilated convolutions maintaining high-resolution representations. The hybrid models mitigated the computational burden of maintaining dual encoders because their convolutional branches produced compact feature maps at the final stage, allowing decoders to operate on smaller spatial inputs. Consequently, all four hybrid architectures operated faster than the convolutional baseline on San Donaci, with Hybrid 4 recording an inference time of 0.0903 seconds per image. However, on the higher-resolution Zeuli dataset, dual-encoder latency increased, with Hybrid 4 requiring 0.3860 seconds per frame compared to 0.2432 seconds for baseline SegFormer and 0.3037 seconds for DeepLabV3+.
The study demonstrates that explainable artificial intelligence can serve as an active intermediary in neural network development rather than merely functioning as an explanatory endpoint. By grounding architectural hypotheses in empirical measurements of how separate networks allocate visual attention, roboticists and computer vision engineers can deliberately combine complementary feature representations to tackle complex agricultural perception challenges. The authors noted that deploying such dual-encoder segmentation systems on resource-constrained robotic platforms will necessitate future research into post-training network optimization, including weight quantization and structural pruning, alongside broader evaluations across diverse geographic growing regions, training configurations, and crop phenological stages.
Subject of Research: Explainable AI-guided hybrid deep learning for grapevine semantic segmentation
Article Title: XAI-guided development of hybrid CNN-transformer architectures for grapevine semantic segmentation
Article References: Bono, A., Guaragnella, C., & D’Orazio, T. (2026). XAI-guided development of hybrid CNN-transformer architectures for grapevine semantic segmentation. Smart Agricultural Technology, 15, Article 102708. https://doi.org/10.1016/j.atech.2026.102708
Image Credits: AI Generated
DOI: 10.1016/j.atech.2026.102708
Keywords: semantic segmentation, precision viticulture, explainable artificial intelligence, Grad-CAM, convolutional neural networks, vision transformers, DeepLabV3+, SegFormer, EfficientNet-B4, MiT-B2, hybrid neural networks, grapevine canopy, Pointing Game, computer vision, agricultural robotics, cross-attention
News Source: Blake Davidson. (November 13, 2026). Explainable Artificial Intelligence Directs Hybrid Neural Networks for Precise Vineyard Component Segmentation. Scienmag.
