One Click Picks the Fruit: Robot Harvesting Learns New Targets from a Single Image

A harvesting robot rolling into an unfamiliar orchard has traditionally faced a hard problem: its vision system was trained on a fixed set of crops, and recognizing anything else meant collecting thousands of images, annotating them by hand, and retraining the model. The researchers, led by Junyu Hao and colleagues, present a one-shot object detection method in which an orchard worker simply clicks on one example of the target fruit in a single image, and the robot then finds every matching fruit in subsequent scenes without any additional training or annotation.

The workflow unfolds in four stages. First, the system gathers the information it needs: the support image from the worker’s viewpoint, the click or selection marking the target, and a live query image captured from the robotic arm’s camera. Next, a support object generation module, built on the pre-trained image encoder, mask decoder, and prompt encoder of the Segment Anything-style architecture by Kirillov and colleagues, converts that single click into an accurate segmentation mask of the chosen fruit. In parallel, a candidate generation module applies zero-shot segmentation to the query image, proposing all plausible object regions, which are then filtered using the aspect ratio of the support object as a screening condition.

The heart of the method is feature matching. A deep Siamese network based on an autoencoder encodes both the support object and each candidate region into 1024-dimensional feature vectors. The authors evaluated ResNet-18, ResNet-34, and ResNet-50 backbones for this encoder and selected ResNet-50 because it achieved the highest peak signal-to-noise ratio and structural similarity and the lowest mean squared error in image reconstruction tests. A three-level multilayer perceptron then compresses these features to 512, 256, and 128 dimensions, capturing differences from coarse to fine scales, and similarity is judged by the Euclidean distance between embedding pairs. If that distance falls below a threshold of 0.71, searched over the interval from 0 to 1 in steps of 0.01, the candidate is accepted as the target class.

Detection is not merely a yes-or-no decision. A separate refinement network, trained with a mean-squared-error loss on position offsets, predicts the shifts in center coordinates, width, and height needed to move each coarse proposal to a precise bounding box. During training, a contrastive loss pulls features of same-class image pairs together and pushes different-class pairs apart beyond a set margin, and a joint loss combines this with the offset regression. The ablation study shows both components matter: adding the autoencoder raised mean average precision by 10.57 points, and the two position fine-tuning constraints contributed further gains, bringing the full system to 29.93 mAP on the benchmark.

That benchmark is the COCO dataset, evaluated under a standard one-shot protocol in which 80 categories are split into 60 base classes for training and 20 novel classes for testing, with only bounding-box annotations used. Against recent one-shot methods including Halluc, SFOT, Norm-VAE, SNIDA, MPAD, and MMKT, the proposed approach reached 29.93 mAP, 37.99 AP50, and 32.28 AP75, the strongest figures among the compared methods.

The more agriculturally relevant test was cross-fruit detection. The team assembled datasets from real harvesting environments: 1100 kumquat images collected in Guangxi, 890 apple images from Beijing, 900 cluster tomato images from a commercial farm in Haidian District, plus the public StrawDI_Db1 strawberry dataset from Huelva, Spain, and a pear dataset from the University of Tsukuba in Japan. Crucially, the proposed model was pretrained only on COCO and received no fine-tuning on any of these fruit datasets, while the fully supervised baseline, YOLO11n, was trained directly on the target categories. Despite that handicap, the one-shot method achieved 57.78 mAP, 79.10 AP50, and 48.72 AP75, exceeding YOLO11n by 4.00, 7.07, and 2.53 percentage points respectively, and clearly outperforming other exemplar-based detectors such as Siamese-RPN, OS2D, and MQ-Det.

Robustness testing probed four challenging conditions drawn from real harvesting scenes: dense occlusion, shadows, motion blur, and very small fruits. Under dense occlusion, mean average precision dropped to 51.87 percent, 5.91 points below the overall result, and shadow conditions barely affected performance at 57.46 percent. Motion blur, simulated with a linear kernel of 25 pixels because too few naturally blurred samples existed, reduced mAP to 46.85 percent. Small fruits proved hardest of all at 38.92 percent mAP, since such targets contain little visual information, though the authors note the method still detects a substantial share of them.

The system was also deployed on an integrated harvesting robot equipped with an Intel RealSense D435i camera and a robotic arm, using a UR5 arm for kumquat harvesting and an FR5 arm for apples. Across the full workflow of image acquisition, recognition, and grasping, the method recognized 216 of 236 kumquat objects, a 91.53 percent recognition rate, and 105 of 109 apple objects, or 96.33 percent. Computational measurements on an NVIDIA RTX 4090 desktop GPU showed an average processing time of 34.42 milliseconds per image, roughly 29.06 frames per second, with peak GPU memory use of just 0.24 gigabytes, although the authors acknowledge that embedded deployment will require future model compression and hardware acceleration.

The authors are candid about limits. The method is designed for individual-fruit instance detection: while individual tomatoes within a cluster can be found, recognizing an entire cluster as one harvesting unit remains difficult. Performance also degrades under occlusion, blur, and small target scales, and false detections in agriculture carry real costs in yield losses, which is one reason the team avoided text-prompt-driven zero-shot detectors, whose semantic ambiguity and parsing errors they consider too fragile for the field.

Even so, the study sketches a practical path toward robots that adapt as quickly as orchards change. When a new cultivar, maturity target, or orchard arrives, the worker specifies the target once with a click, and no repeated human intervention is needed afterward. In scenarios where rebuilding a fully supervised dataset is impractical, the demonstrated combination of zero-shot segmentation, Siamese feature matching, and position refinement offers harvesting robots a way to be useful on day one in places their training data never described.

Subject of Research: One-shot object detection for robotic orchard harvesting using a single user-specified reference image

Article Title: User-friendly orchard harvesting object detection through easy interaction

Article References: Hao, J., Yang, M., Wei, J., Pan, H., Chen, J., Zhan, Y., Zheng, J., & Yang, M. (2026). User-friendly orchard harvesting object detection through easy interaction. Smart Agricultural Technology, 15, Article 102578. https://doi.org/10.1016/j.atech.2026.102578

Image Credits: AI Generated

DOI: 10.1016/j.atech.2026.102578

Keywords: one-shot object detection, harvesting robots, orchard automation, zero-shot segmentation, Siamese network, computer vision, agricultural robotics, fruit detection, deep learning, COCO benchmark, robotic grasping, precision agriculture