Dual-Branch Network Segments Grapevine Stems and Branches for Robotic Vineyard Pruning

Agricultural engineers have developed a dual-branch deep learning network designed to segment dormant grapevine stems and branches from complex vineyard imagery, providing a dedicated computer vision pipeline for automated winter pruning. Grapevine pruning serves as a foundational viticultural practice that regulates vegetative growth, fruit production, vine vigor, and overall grape quality. In conventional vineyard operations, this seasonal task relies almost entirely on manual field labor, which requires experienced agricultural workers and introduces substantial operational expenses, variable pruning quality, and low operational efficiency across large acreages. While vision-guided agricultural robotics has been proposed to overcome seasonal labor shortages and standardize vine management, reliably segmenting thin, low-contrast branches against cluttered natural vineyard backdrops remains a challenging perception problem. To address these spatial and structural complexities, the newly introduced architecture unites vision transformer global context modeling with discrete wavelet transforms and state-space sequence modeling in a synchronized dual-encoder framework.

The computer vision model, designated as the Dual-branch Transformer-Mamba Fusion Network, or DTMF, addresses the inherent limitations of employing single-paradigm neural architectures for outdoor agricultural scene parsing. Standard convolutional neural networks and pure vision transformer backbones often struggle to simultaneously capture extensive spatial dependencies and fine-grained frequency-domain characteristics, particularly when branch boundaries blend into soil, trellis wires, or ambient vegetation. Within the DTMF architecture, processing is distributed across two complementary subnetworks designated U1 and U2. Subnetwork U1 employs a Vision Transformer encoder to capture broad contextual relationships across the entire vineyard scene, utilizing stacked multi-head self-attention mechanisms and multilayer perceptrons after partitioning input imagery into embedded patches. In parallel, subnetwork U2 incorporates a discrete wavelet transform and Mamba fusion encoder to isolate multi-scale spatial and frequency components, ensuring that high-frequency edge textures and low-frequency structural surfaces are processed through dedicated analytical pathways.

To coordinate the interaction between these disparate feature domains, the research team designed a specialized Cross-scale Wavelet-Mamba Fusion module within the frequency-processing encoder. Input feature maps entering this module are initially decomposed through a 2D discrete wavelet transform into distinct low-frequency approximations and high-frequency directional sub-bands. The low-frequency components, which represent smooth stem surfaces and macroscopic plant geometry, are concatenated with upsampled intermediate features from the Vision Transformer context encoder and routed through Mamba state-space blocks to model long-range sequential continuity with linear computational complexity. Simultaneously, the high-frequency sub-bands, which encapsulate slender branch outlines and intricate bark textures, undergo focused refinement through a dedicated self-attention module. The resulting feature streams are subsequently unified through a Perception-Guided Attention and Feedforward module, which dynamically modulates attention weights using cross-scale query, key, and value mappings before passing through non-linear convolutional feedforward layers and residual connections.

The decoding phase of DTMF mirrors the dual organization of its front end by deploying two separate decoding paths rather than prematurely collapsing features into a single reconstruction trunk. The U1 decoder follows a classic U-Net formulation, progressively restoring spatial dimensions via transposed convolutions while receiving skip connections directly from the Vision Transformer encoder blocks. In contrast, the U2 decoder integrates Mamba sequence blocks paired with specialized edge feature enhancement modules, accepting refined multi-scale skip connections directly from the wavelet-mamba encoder to reconstruct fine branch margins. These two independent decoder outputs remain separated until the final decision stage, where their respective soft prediction tensors are merged via a weighted fusion operator before applying an argmax decision rule to produce the final discrete semantic mask distinguishing foreground stem and branch structures from background elements.

Model training was governed by an adaptive, two-stage composite objective function structured to balance regional mask convergence during early iterations with localized contour refinement during later epochs. During the initial training phase spanning the first four epochs, the overall loss combined standard pixel-level binary cross-entropy, an overlap-maximizing Dice loss, and an inter-branch hybrid supervision loss that penalizes discrepancies between the individual soft predictions of the two decoders and their combined hard target. Beginning at epoch five, the binary cross-entropy component was entirely substituted with a superpixel loss formulated over Simple Linear Iterative Clustering regions. This region-based loss explicitly penalizes inconsistent class predictions within localized superpixel clusters of uniform color and texture, compelling the optimization process to conform tightly to the natural physical boundaries of slender canes without sacrificing global classification stability.

The empirical evaluation of DTMF was conducted on an experimental image dataset comprising 2,150 high-resolution RGB photographs collected during the dormant winter season across two distinct viticultural regions in Xinjiang, China. The acquisition campaign gathered 1,250 images in November 2024 from an experimental vineyard at Shihezi University and 900 images in December 2024 from a commercial production orchard in Turpan. All imagery was captured under ambient outdoor lighting using a Sony digital camera fitted with an 18 to 150 millimeter zoom lens positioned at varying angles and distances to encompass diverse vine architectures, trellis interferences, and ground conditions. The dataset was partitioned into 1,505 training images, 215 validation images, and 430 dedicated test images, with data augmentation techniques including random cropping, horizontal and vertical translations, Gaussian noise injection, and rotational shifts within a range of 30 degrees expanding the effective training volume to 4,515 images.

Every image across the training, validation, and test splits received meticulous manual pixel-level annotations delineating foreground stems and pruning branches using Labelme software. The 430 test images were reserved strictly for final performance benchmarking and remained completely uninvolved in model training or validation checkpoint selection. Quantitative evaluation followed a rigorous, unified three-class evaluation protocol covering background, stem, and branch categories. To ensure analytical consistency, pixel predictions across all test scenes were accumulated into a single pooled three-by-three integer confusion matrix, from which mean Pixel Accuracy, mean Intersection over Union, and individual class IoU metrics were directly calculated. The final complete DTMF network achieved a mean Pixel Accuracy of 94.24 percent and a mean Intersection over Union of 88.25 percent across the evaluated test cohort.

Detailed comparative benchmarking against established semantic segmentation architectures demonstrated clear quantitative differences, particularly in the fine-branch category where conventional baselines frequently suffer from broken masks and canopy-edge omissions. In overall test evaluations, DTMF recorded an mIoU of 88.25 percent, surpassing DeepLabV3 at 69.65 percent, PSPNet at 60.53 percent, standard FCN at 72.20 percent, Segformer at 68.12 percent, baseline U-Net at 72.03 percent, and a Transformer-backbone U-Net at 73.12 percent. Class-specific breakdowns revealed that while stem segmentation was relatively competitive across modern models—with DTMF reaching 87.68 percent stem IoU compared to 87.46 percent for the Transformer-equipped U-Net—the segmentation of slender branches exhibited a substantial performance gap. DTMF achieved an 85.12 percent branch IoU, representing a 14.99 percentage point improvement over the strongest baseline branch score of 70.13 percent recorded by FCN.

Ablation experiments confirmed the complementary contributions of the dual-pathway layout, the wavelet-mamba fusion modules, and the two-stage loss schedule. Isolating the Transformer encoder and its paired decoder established a baseline mPA of 81.11 percent and mIoU of 71.98 percent, while deploying the discrete wavelet transform and Mamba encoder alone produced modest gains to 81.74 percent mPA and 74.00 percent mIoU. Merging both encoders into a single Mamba decoder yielded 85.73 percent mPA and 80.65 percent mIoU, whereas deploying the complete dual-encoder and dual-decoder configuration elevated performance to the peak 94.24 percent mPA and 88.25 percent mIoU. Similarly, component-level ablations within the fusion module revealed that adding Mamba blocks to the initial wavelet baseline raised mIoU from 80.32 percent to 83.65 percent, and incorporating high-frequency self-attention further advanced mIoU to 86.65 percent. Loss-function trials likewise showed that the two-stage schedule incorporating hybrid supervision and superpixel constraints improved branch IoU from 74.33 percent under basic Dice and cross-entropy training to 85.12 percent.

Qualitative inspections and class activation mappings further illustrated the visual behavior of the network across complex occlusions and externally sourced images of woody plants. Class activation heatmaps indicated that integrating discrete wavelet decomposition with Mamba sequence modeling produced more continuous spatial activation traces along primary canes and secondary lateral branches compared to configurations omitting these modules. In comparative visual samples, the full model accurately tracked thin canes across background wires and irregular soil textures with minimal over-segmentation or under-segmentation artifacts. Additional qualitative trials on external woody-plant images from third-party repositories demonstrated structural mask delineation on unseen tree canopies, though the authors emphasize that these demonstrations remain purely illustrative and do not constitute quantified domain generalization or transfer verification.

The investigators noted several key limitations inherent in the current experimental design and computational framework. First, the parallel dual-encoder and dual-decoder architecture inevitably increases parameter count and computational complexity, posing potential latency challenges for deployment on resource-constrained embedded computing platforms aboard vineyard robotics. Second, the empirical dataset was collected exclusively during the dormant winter season within two geographic regions in Xinjiang, leaving cross-season robustness against summer foliage and varied regional trellis structures unverified. Third, the study utilized an image-level split rather than plant-level grouping, and repeated-seed training trials were not conducted to quantify cross-run variance. Finally, while accurate pixel-level semantic segmentation is a necessary perception prerequisite, practical robotic pruning requires subsequent three-dimensional skeletonization, cutting-point localization algorithms, geometric path planning, and physical end-effector manipulation.

Future viticultural engineering efforts will aim to optimize the computational efficiency of the framework while extending its functional capabilities toward closed-loop field deployment. The authors highlight structured parameter pruning, network quantization, and knowledge distillation—such as training a lightweight, wavelet-guided Mamba student model from the dual-branch teacher—as viable avenues to reduce inference latency without degrading segmentation fidelity. Additional planned investigations involve integrating explicit topology-aware loss functions to preserve continuous cane connectivity, incorporating multimodal perception inputs such as stereoscopic depth maps and temporal video sequences, and exploring test-time domain adaptation to minimize reliance on intensive manual pixel-level labeling across diverse vineyard environments.

Subject of Research: Deep learning semantic segmentation for automated grapevine pruning perception

Article Title: A dual-branch transformer-mamba network integrating cross-scale wavelet-mamba fusion for semantic segmentation of grapevine stems and branches

Article References: Song, L., Song, Z., He, L., Lyu, X., Bao, P., & Shen, C. (2026). A dual-branch transformer-mamba network integrating cross-scale wavelet-mamba fusion for semantic segmentation of grapevine stems and branches. Smart Agricultural Technology, 15, Article 102651. https://doi.org/10.1016/j.atech.2026.102651

Image Credits: AI Generated

DOI: 10.1016/j.atech.2026.102651

Keywords: Grapevine Pruning, Semantic Segmentation, Computer Vision, Vision Transformer, Mamba Architecture, Discrete Wavelet Transform, Agricultural Robotics, Precision Viticulture, Feature Fusion, Superpixel Loss, Dual-Branch Network, Image Segmentation, Dormant Season, Deep Learning, Viticulture Automation, State-Space Models