AI pasture models stumble when tested in states they never saw

Deep learning models that promise to weigh pastures from photographs may be far less trustworthy than their published accuracy figures suggest. A new open-access study in Discover Artificial Intelligence has put two leading convolutional neural network architectures through one of the most demanding validation regimes yet applied to agricultural computer vision, and the results expose a sobering gap between how well these models perform on familiar ground and how badly they can fail when moved to a state they have never seen. The work, led by Jesus Rafael Hechavarria-Hernandez of Universidad Ecotec in Ecuador together with Rubén Dario Ramos-Ciprián and Rafael Omar Batista-Jorge in the Dominican Republic, combined a multi-seed leave-one-state-out validation framework with a systematic explainable AI audit to probe exactly where and why image-based biomass prediction breaks down under spatial domain shift.

The stakes are considerable. Precision agriculture increasingly relies on deep learning models fused with satellite and drone imagery to estimate agroecological variables such as plant biomass, and published studies routinely report near-perfect coefficients of determination, sometimes exceeding 0.90, for pasture monitoring from unmanned aerial vehicle data. Neural networks have been shown to clearly outperform Random Forest and Support Vector Machine baselines when trained and tested on high-resolution UAV imagery, with one comparison reporting a median root mean squared error roughly fifty percent lower for the neural approach. But these headline figures are typically produced under relatively homogeneous or controlled conditions, where the training and test data come from the same distribution. In the real world, soil type, vegetation species, foliage coloration, and cultivation patterns vary drastically between regions, and a model trained in one limited context can encounter a domain shift so pronounced that extrapolating it to a new environment becomes extremely challenging.

To simulate this deployment scenario rigorously, the researchers turned to the CSIRO Image2Biomass Prediction dataset, a benchmark hosted on Kaggle containing 357 unique pasture samples from four Australian states: New South Wales, Tasmania, Victoria, and Western Australia. Each sample pairs a UAV photograph with metadata such as state, species, and pre-acquisition NDVI, together with five regression targets covering dry clover, dry dead, dry green, dry total, and green dry matter biomass in grams. The geographic distribution is highly imbalanced, with Western Australia contributing only nine percent of the data, or 32 samples, and biomass distributions vary significantly across state lines, indicating high visual and ecological heterogeneity. Standard random k-fold cross-validation, which mixes data from all regions, would likely produce overly optimistic performance estimates precisely because the model is evaluated on distributions similar to those of its training set.

The validation protocol adopted instead was Leave-One-State-Out, or LOSO, cross-validation, in which one state is completely excluded from training and used exclusively as an unseen test domain. The remaining three states form the training-validation set, within which an internal eighty-twenty split is employed for model selection and early stopping. Crucially, the researchers repeated the entire LOSO procedure across three independent random seeds, 42, 123, and 2025, and two convolutional neural network architectures, ResNet50 and EfficientNet-B0, yielding twenty-four independent deep learning experiments in total. Both architectures were initialised with ImageNet-pretrained weights and adapted for multi-output regression through custom heads comprising a dropout layer, a linear mapping to the five biomass targets, and a Softplus activation to constrain predictions to non-negative values. All numerical metadata variables were deliberately excluded from the deep learning training process, forcing the CNNs to rely exclusively on RGB imagery. Training used the AdamW optimizer with a learning rate of one times ten to the minus four and a weight decay of the same magnitude, a batch size of sixteen, and early stopping with a patience of five epochs, with the checkpoint achieving the minimum validation WRMSLE retained for final evaluation rather than the model state from the final training epoch.

The primary performance metric was the Weighted Root Mean Squared Logarithmic Error, or WRMSLE, chosen because its logarithmic formulation emphasises relative prediction accuracy while attenuating the influence of large absolute errors. The training objective itself minimised mean squared error in log-transformed target space, ensuring that all five biomass components contributed equally to the gradient-based optimisation. Performance was assessed using WRMSLE, Mean Absolute Error, Root Mean Squared Error, and the coefficient of determination, with seed-to-seed training variability summarised through standard deviations across the three random seeds and ninety-five percent bootstrap confidence intervals for test WRMSLE estimated by resampling the held-out test samples within each seed-level LOSO run. The researchers are careful to note that these bootstrap bounds summarise test-sample resampling uncertainty within fixed held-out domains and should not be interpreted as population-level confidence intervals for geographical domain generalization.

The results reveal a stark generalization gap. ResNet50 achieved mean test WRMSLE values ranging from 0.869 in Tasmania to 1.460 in Western Australia, with intermediate values of 0.889 in Victoria and 1.321 in New South Wales. EfficientNet-B0 followed a similar pattern, ranging from 0.911 in Tasmania to 1.519 in Western Australia, with 0.923 in Victoria and 1.484 in New South Wales. The largest errors consistently occurred in the geographically held-out NSW and WA folds for both architectures, while validation errors remained relatively low and stable across folds. More striking still, the coefficient of determination for the primary dry total biomass target produced negative mean test values across all four held-out domains for EfficientNet-B0, reaching minus 2.046 in New South Wales, minus 0.948 in Tasmania, minus 1.216 in Victoria, and minus 0.323 in Western Australia. ResNet50 also yielded negative mean test R-squared values in New South Wales, minus 1.359, and Victoria, minus 0.394, while producing values close to or above zero in Tasmania, 0.014, and Western Australia, 0.290. Negative R-squared values indicate that, under the corresponding LOSO conditions, model predictions were less accurate than a constant predictor based simply on the mean target value of the held-out domain.

Perhaps the most surprising finding came from the comparison against a metadata-based Random Forest baseline trained exclusively on non-image variables available at inference time, including species information, vegetation height, NDVI measurements, and acquisition month. Despite having no access to UAV imagery and therefore no ability to exploit visual patterns in the photographs, the baseline achieved competitive performance in all states and outperformed both deep learning architectures in New South Wales and Western Australia, the two domains exhibiting the strongest generalization failures. In New South Wales, the Random Forest model achieved a test WRMSLE of 1.128, compared with 1.321 for ResNet50 and 1.484 for EfficientNet-B0. Similarly, in Western Australia, the baseline obtained a WRMSLE of 1.366, outperforming both ResNet50 at 1.460 and EfficientNet-B0 at 1.519. In Tasmania and Victoria, the differences between the baseline and the CNN models were comparatively small. These findings suggest that a substantial fraction of the predictive signal may be captured through relatively simple ecological descriptors rather than exclusively through image-derived features, and that the CNN models failed to consistently learn domain-invariant representations capable of generalising across geographical regions.

To investigate the mechanisms underlying these failures, the researchers conducted a systematic explainable AI audit based on Grad-CAM, or Gradient-Weighted Class Activation Mapping, a post-hoc technique that visualises the image regions contributing most strongly to a model’s prediction. Unlike conventional XAI demonstrations that rely on a handful of manually selected examples, this study analysed forty Grad-CAM activation maps generated from the LOSO evaluation protocol, selecting for each geographical domain the five samples with the lowest prediction errors and the five with the highest errors based on the absolute error of the dry total biomass target. The audit was performed using ResNet50 with the reference seed of 42, with activation maps generated from the final convolutional block and superimposed onto the original RGB images. A quantitative vegetation-attention proxy was also computed by measuring the proportion of Grad-CAM activation located over vegetation-dominated pixels, though the researchers caution that this proxy is partially constrained by the proportion of vegetation present in each image and should be interpreted as a coarse diagnostic indicator rather than a ground-truth segmentation metric.

The Grad-CAM audit revealed two complementary failure mechanisms. Across all four domains, the worst-performing predictions consistently exhibited higher vegetation-attention ratios than the best-performing predictions, with the mean vegetation attention in Victoria increasing from 0.878 for the best predictions to 0.952 for the worst, in Tasmania from 0.427 to 0.720, and in Western Australia from 0.387 to 0.784. Yet high levels of vegetation attention were not necessarily associated with low prediction error. In several cases, the model focused strongly on vegetation regions while still producing substantial errors, suggesting that the issue was not merely where the model was looking but how it interpreted the visual information present in those regions. In New South Wales, the activation map for the highest-error sample, which carried a prediction error of 161.5 grams, displayed a relatively diffuse attention pattern distributed across multiple vegetation structures, which the researchers describe as an attentional wash indicative of semantic uncertainty in which the model was unable to construct a stable representation of vegetation under unfamiliar visual conditions. In Tasmania and Victoria, attention concentrated on dense vegetation regions, suggesting that the model correctly identified biologically relevant areas despite producing substantial prediction errors.

In contrast, the Western Australia sample exhibited a highly localised activation pattern concentrated on background structures and soil-related textures rather than on the vegetation itself. Such behaviour is consistent with shortcut learning, whereby a model relies on spurious contextual correlations acquired during training rather than on biologically meaningful biomass indicators, a phenomenon resembling the well-known Clever Hans effect in which machine-learning systems exploit superficial correlations that are predictive within the training distribution but lack causal relevance to the underlying task. The persistence of these domain-dependent attention patterns despite the applied augmentation strategy, which included random horizontal flips, random vertical flips, and random rotations of up to plus or minus ten degrees, suggests that although these transformations likely improved training robustness as a regularisation mechanism, they were insufficient to eliminate reliance on contextual visual cues under geographically independent evaluation. Taken together, the LOSO evaluation, the baseline comparison, and the XAI audit converge toward the same conclusion: the primary limitation of the evaluated CNN models is not predictive capacity within known domains but robustness under spatial distribution shifts.

The researchers are careful to acknowledge several limitations that constrain broader inference. Only three independent random seeds were evaluated, limiting the precision with which seed-to-seed training variability can be characterised. The study included only four geographical domains, and domain identity was partly confounded with sample size and representativeness, particularly for Western Australia, which was both the smallest held-out fold and one of the most difficult domains, meaning that its elevated error may reflect a combination of ecological distinctiveness, limited visual coverage, and small-sample instability rather than geographical domain shift alone. Moreover, although LOSO prevented information from the held-out state from entering the training or validation sets, the available metadata did not permit paddock-level, field-level, or acquisition-site grouping within the three training domains, so spatially correlated observations may have occurred in both the internal training and validation subsets, potentially inflating internal validation performance and exaggerating the apparent magnitude of the validation-test gap. The reported gap should therefore be interpreted as a combined signal of cross-domain transfer difficulty and potential within-state validation optimism rather than as a pure estimate of domain-shift effects.

The principal practical implication is that a pasture biomass model should not be considered deployment-ready for a new geographical region solely because it performs well under internal validation. Spatially independent testing, comparison with strong metadata baselines, and explainability-based auditing are required before operational use, especially in fields where management decisions such as stocking rates or fertilisation carry economic and environmental stakes and farmers and agronomists require transparency to trust algorithmic outputs. Beyond the specific application investigated here, the proposed combination of multi-seed LOSO validation, strong non-image baselines, bootstrap-based uncertainty analysis, and systematic XAI auditing constitutes a transferable evaluation framework for assessing the reliability of agricultural computer-vision systems operating under spatial uncertainty. Future work should investigate XAI-informed data augmentation, where Grad-CAM diagnostics guide the generation of training samples that explicitly reproduce failure contexts such as domain-specific soil textures and illumination patterns, alongside domain generalization architectures including Domain-Adversarial Neural Networks, vegetation-normalized attention metrics that explicitly account for scene composition, and multimodal learning strategies that integrate UAV imagery with structured ecological variables such as NDVI, vegetation height, and species composition. These complementary approaches may reduce shortcut-learning behaviour and improve the reliability of agricultural AI systems operating across heterogeneous environmental conditions, moving the field toward computer-vision systems that are not only accurate within known environments but also robust, interpretable, and transferable across previously unseen geographical domains.

Subject of Research: Domain generalization testing of deep learning models for pasture biomass estimation

Article Title: Evaluating domain generalization in agricultural deep learning through multi-seed LOSO validation and explainable AI

Article References: Hechavarria-Hernandez, J. R., Ramos-Ciprián, R. D., & Batista-Jorge, R. O. (2026). Evaluating domain generalization in agricultural deep learning through multi-seed LOSO validation and explainable AI. Discover Artificial Intelligence, 6(1), Article 1524. https://doi.org/10.1007/s44163-026-02545-w

Image Credits: AI Generated

DOI: 10.1007/s44163-026-02545-w

Keywords: domain generalization, deep learning, pasture biomass estimation, leave-one-state-out validation, Grad-CAM, explainable AI, precision agriculture, spatial domain shift, ResNet50, EfficientNet-B0, Random Forest baseline, CSIRO Image2Biomass dataset