Potato Leaf AI Study Reveals Visual Similarity Is Not Leaf Identity

A team of Peruvian computer scientists has reevaluated how deep learning models should be tested on potato leaf disease images, and their findings strike at a subtle but consequential assumption embedded in much of plant pathology AI research: that images which look alike must be pictures of the same leaf. In a study published in Frontiers in Computer Science, researchers led by Hugo Vega-Huerta of Universidad Nacional Mayor de San Marcos ran 240 complete training runs across four neural architectures, two datasets, and fifteen random seeds, and showed that the way images are divided between training and testing can shift measured performance by several percentage points. Their central methodological message is carefully bounded: visual similarity in a modern self-supervised embedding space does not, by itself, establish that two photographs show the same physical specimen, and performance drops after regrouping data reflect a change in the evaluation unit rather than proof that duplicated images alone caused the decline.

The laboratory portion of the study drew on the potato subset of the widely used PlantVillage collection, comprising 2,152 images: 152 healthy leaves, 1,000 with early blight, and 1,000 with late blight. Crucially, the researchers did not treat these images as independent observations. They exploited a published PlantVillage metadata file that maps every image to one of 538 leaf identities, each represented by four photographs. This external reference allowed them to construct holdout partitions in which no leaf appearing in training also appeared in testing. The field dataset, by contrast, contained 1,296 images from a single Kaggle source, labeled Healthy, Fungi, or Phytophthora, with no published identity metadata available. Because no human annotations of near-duplicates existed for the field images, the team grouped them using visual clusters derived from DINOv2, a pretrained vision model that produces rich image embeddings without task-specific training.

That choice required validation, and the validation delivered a sobering result. When the researchers compared DINOv2 clusters, formed at a cosine-distance threshold of 0.10, against the published laboratory leaf identities, agreement was poor. Of the 41,381 pairs of images placed in the same cluster, only 1,611 shared a published leaf identity, yielding a pair precision of just 3.89 percent. Pair recall reached 49.91 percent and weighted identity purity 38.34 percent. Raising the threshold to 0.15 or 0.20 increased recall to 89.22 and 95.82 percent respectively, but precision collapsed to 0.40 and 0.18 percent as clusters ballooned to encompass thousands of unrelated image pairs. The authors are explicit that this does not validate DINOv2 as a duplicate detector; it demonstrates that images sharing a visual cluster cannot be assumed to be repeated views of the same leaf, a distinction with direct consequences for how grouped evaluations should be interpreted.

The exposure audit quantified how much the splitting strategy matters. Under a conventional stratified random holdout on the laboratory dataset, a mean of 97.13 percent of test images shared a leaf identity with at least one training image, meaning the model had almost always seen other photographs of the very same leaf during training. Under identity-grouped holdout, that exposure fell to zero in all fifteen seeds. Byte-identical file exposure, measured with SHA-256 hashes, was zero under both laboratory strategies, indicating that outright file duplication was not the issue; the leakage operated at the level of shared specimens. In the field dataset, exact hash exposure averaged 4.48 percent under random splitting and zero under grouped splitting, while 41.30 percent of test images shared a visual cluster with training images under random assignment. The authors caution that zero hash overlap does not exclude recompressed or transformed copies, and that field group exposure should not be read as the prevalence of confirmed near-duplicates.

To make the two splitting strategies comparable, the team used an integer-programming assignment that preserved whole groups while targeting the same overall and per-class allocations as the random splits, with overall proportions required to fall within 0.02 of targets and per-class allocations within 0.03. Four architectures were fine-tuned under identical conditions: ResNet50, MobileNetV3 Large, EfficientNetV2 S, and ViT B/16, each trained for ten epochs with batch size 32, an AdamW learning rate of 0.0003, and unweighted cross-entropy loss. The checkpoint with the highest validation macro F1 was selected for testing, and test results were never used for model selection. Every one of the 240 runs completed, and the audit verified 62,160 prediction records, 240 training histories, 60 partition files, and file checksums, with recomputed metrics matching the archived results exactly.

The performance gaps were consistent in direction, though variable in size. In the laboratory dataset, accuracy gaps between random and grouped splits ranged from 0.16 to 1.00 percentage points, while balanced accuracy gaps ranged from 0.27 to 4.03 points. ViT B/16 showed the largest laboratory balanced gap at 4.03 points, and ResNet50 also showed a positive balanced effect after statistical adjustment. In the field dataset the effects were larger: accuracy gaps ranged from 4.45 to 6.25 points, balanced accuracy from 5.31 to 9.54 points, and macro F1 from 4.50 to 7.50 points. MobileNetV3 exhibited the largest field gap on all three metrics, with balanced accuracy falling from a mean of 85.90 percent under random splitting to 76.37 percent under visual-group splitting. The pattern, the authors stress, is therefore not confined to transformer architectures with global attention mechanisms.

Statistical rigor shaped which of these differences the team was willing to highlight. Using Benjamini-Hochberg correction across the complete family of 24 comparisons, four models, two datasets, and three metrics, seventeen comparisons met the false discovery rate criterion of q below 0.05. Notably, all twelve field comparisons passed, along with laboratory ResNet50 balanced accuracy and macro F1 and all three laboratory ViT metrics. The authors also ran an exploratory interaction contrast, subtracting the mean split gap of the three convolutional networks from the ViT gap within each seed, to test whether transformers are disproportionately sensitive to grouping. The balanced-accuracy contrast was positive at 2.91 points in the laboratory and negative at 3.18 points in the field, but neither survived adjustment across the six tests, with q-values of 0.2211 in both cases. The researchers explicitly frame this as absence of evidence for an architectural difference, not evidence that convolutional networks and vision transformers are equivalent in leakage susceptibility.

The study is equally candid about its own limitations, several of which stem from its history as a correction of earlier work. The analysis is a post hoc reconstruction following review of prior results, not a preregistered replication, and historical grouped and mixed-source field results were excluded from the main analysis. The 0.10 clustering threshold for the field data is a declared heuristic adopted during revision, not an independently calibrated duplicate detector, and no threshold-performance sweep was performed. Field near-duplicates have never been labeled by human reviewers, so the visual grouping remains exploratory. The laboratory reference itself is published metadata rather than independently reannotated ground truth. Test sets contained only 23 or 24 healthy laboratory images and 30 healthy field images, making class-specific estimates unstable, and the fifteen repeated seeds describe protocol variability within fixed corpora rather than fifteen independent biological cohorts. Uncertainty was conditional on these datasets and did not represent replication across independent farms or populations.

What the study firmly establishes is methodological rather than causal. The unit of independence, whether the published leaf identity in the laboratory or the DINOv2 visual cluster in the field, materially changes measured performance on these datasets, and visual similarity alone cannot establish specimen identity. The authors recommend that future plant disease classification studies specify their unit of independence explicitly, audit exposure against an appropriate reference, retain per-image predictions for exact reanalysis, and report balanced metrics alongside uncertainty intervals, since a small overall accuracy difference can coexist with a much larger balanced-accuracy gap when errors concentrate in a minority class. They also warn that high accuracy on controlled or single-source collections should not be interpreted as clinical, agronomic, or deployment-level performance. Before visual groups in field data can be described as confirmed near-duplicate groups, human annotation of field specimens remains an unresolved requirement, and the team positions its work as a template for leakage-aware evaluation rather than a verdict on any particular architecture.

Subject of Research: Leakage-aware evaluation of deep learning models for potato leaf disease classification using leaf identity and visual grouping

Article Title: Leaf identity and visual group splitting in potato disease classification

Article References: Guerrero-Jaramillo, A., Vega-Huerta, H., Arredondo-Castillo, G., Enriquez, W., Gil-Calvo, R., Luna-Valdez, J. H., Luna-Figueroa, A. M., Barrera-Cajachagua, M. L., Lázaro-Guillermo, J. C., Chauca, M., Niño-Cueva, M. R., & Adrianzén-Olano, I. (2026). Leaf identity and visual group splitting in potato disease classification. Frontiers in Computer Science, 8, Article 1946705. https://doi.org/10.3389/fcomp.2026.1946705

Image Credits: AI Generated

DOI: 10.3389/fcomp.2026.1946705

Keywords: potato disease classification, data leakage, grouped holdout, DINOv2, PlantVillage, balanced accuracy, specimen identity, vision transformer, convolutional neural networks, evaluation methodology, machine learning, plant pathology