Fractal Basin Geometry Exposes Fundamental Limits to Deep Learning Reproducibility

In an analytical and empirical investigation published in Nature Machine Intelligence, researchers Andrew Ly and Pulin Gong have demonstrated that the persistent difficulty of reproducing and predicting deep learning training outcomes stems from an underlying dynamical systems phenomenon known as riddled basin geometry. Even when computational systems enforce strict determinism by fixing mini-batch ordering on central processing units and eliminating external sources of randomness, gradient descent trajectories frequently arrive at radically different parameter-space destinations and distinct functional solutions. Rather than attributing these discrepancies solely to accumulating numerical noise or hardware variability, the authors show that the geometry of attraction basins governing network optimization can take the form of fat fractals with positive Lebesgue measure. In such systems, every parameter initialization that leads to a specific attractor lies arbitrarily close to initializations that converge to entirely different solutions, establishing a fundamental, scale-insensitive barrier where sharpening initialization precision fails to yield meaningful gains in outcome predictability.

To rigorously uncover the dynamical mechanism generating this unpredictability, the investigators analyzed a minimal neural network model and derived the classical route to riddling from widespread architectural properties. Riddled basins require three primary dynamical ingredients: a chaotic attractor residing within an invariant subspace, a measure-zero set of transversely unstable periodic points embedded within that attractor, and at least one competing attractor located elsewhere in the state space. In neural networks, invariant subspaces naturally arise from intrinsic network symmetries, such as permutation and parity reflections. If the loss function’s transverse curvature is negative or exceptionally large relative to the learning rate, small transverse perturbations away from these periodic points grow over time, ejecting trajectories along heteroclinic pathways toward competing attractors and generating dense hyperwedge structures that perforate the surrounding basin of attraction.

A critical hallmark of riddled basin geometry uncovered in the study is the near-zero parameter-space uncertainty exponent, denoted as phi_p. This quantitative signature reveals that even order-of-magnitude reductions in the scale of initialization perturbations provide only marginal, negligible improvements in training outcome reproducibility, because competing basins remain densely intertwined at arbitrarily fine parameter resolutions.

To determine whether these theoretical insights extend beyond idealized low-dimensional systems, the researchers conducted extensive numerical simulations across deep architectures, including VGGNet, ResNet, and BERT models trained under various optimizers, regularizers, and activation functions. Constrained compute requirements—demanding the simultaneous training of tens of thousands of networks to evaluate basin structures—led to the design of a representative deep network configuration utilizing a tapered VGG-12 network comprising nine convolutional layers and three fully connected layers without biases, containing 12,036 parameters. Trained on the MNIST image classification benchmark corrupted with fifty percent label noise using stochastic gradient descent with momentum, learning rate of 0.1, batch size of 128, and weight decay, the deterministic runs exhibited robust signatures consistent with riddling. Despite the extreme label noise introduced to accelerate stochastic collapse, initializations avoiding the trivial origin attractor converged to an average testing accuracy of 97.6 percent, verifying that the observed fractal phenomena operate within networks capable of strong practical generalization.

The study also details how specific mathematical symmetries in neural network architectures dictate the global structure and invariance of these optimization spaces. For networks equipped with odd activation functions, the authors proved that reflection symmetry holds across affine subspaces associated with sets of vanishing neurons within a given hidden layer, flipping the signs of incoming and outgoing weights while preserving network output. In addition, when architectures possess an even number of hidden layers and omit bias parameters, a complementary transverse reflection symmetry emerges across subspaces where all parameters outside the vanishing set are zeroed, explaining rigorous left-right symmetries observed in destination maps. Permutation symmetries similarly establish invariant subspaces where pairs of neuron weight vectors coincide or exhibit opposite signs. These symmetry-induced manifolds host the chaotic attractors and competing destinations, demonstrating that standard structural design choices in modern deep networks inherently satisfy the formal conditions required for riddling to occur.

To evaluate the stability and contraction properties along these invariant manifolds, the authors implemented the treppen-iteration algorithm using iterative QR decompositions of the Jacobian matrix along deterministic gradient descent paths. For deterministic gradient updates governed by the Jacobian of the identity minus the learning rate multiplied by the Hessian matrix, the infinite-time Lyapunov exponent spectrum was calculated and formally partitioned into longitudinal and transverse components. By structuring the initial orthonormal matrix Q0 with longitudinal basis vectors occupying the initial subspace columns, the linearized dynamics preserve longitudinal alignments, allowing unambiguous separation of transverse expansion rates from longitudinal Lyapunov exponents. In the chaotic transient identified within the permutation-invariant subspace of the minimal model, the converged Lyapunov spectrum confirmed the coexistence of chaotic expansion within the invariant manifold alongside localized transverse instabilities that propel trajectories toward competing invariant subspaces.

Crucially, the authors demonstrated that the consequences of riddled basin geometry transcend internal parameter coordinates and directly govern the functional behavior of trained models. By quantifying functional discrepancy through churn across hundreds of randomly perturbed initialization pairs, the authors fitted power-law relationships to functional similarity measures.

The findings provide a foundational dynamical perspective that complements and broadens the edge of stability framework in machine learning optimization. The researchers observed that as the learning rate increases in the minimal model, the system traverses a sequence of dynamical phase transitions, advancing from unriddled basins to pseudo-riddling containing mixed components, followed by true riddling and transient riddling regimes.

Finally, the authors identified an intriguing operational tension between model performance and training reproducibility, showing that near-zero uncertainty exponents frequently coincide with the optimal learning rates that maximize generalization. In the riddling-like dynamical regime, test generalization error correlated strongly with the functional differences observed between models trained from closely situated initializations, whereas outside this regime functional discrepancies remained near zero and decoupled from generalization performance. By framing deep learning irreproducibility as an inescapable consequence of fat-fractal basin geometry rather than transient hardware or software artifacts, the research establishes a rigorous dynamical framework for understanding optimization limits and guiding future symmetry-breaking interventions in artificial intelligence systems.

Subject of Research: Dynamical systems mechanisms of training unpredictability and riddled basin geometry in deep learning

Article Title: Fractal basin geometry and the limits to predictability and reproducibility in deep learning

Article References: Ly, A., & Gong, P. (2026). Fractal basin geometry and the limits to predictability and reproducibility in deep learning. Nature Machine Intelligence. https://doi.org/10.1038/s42256-026-01336-3

Image Credits: AI Generated

DOI: 10.1038/s42256-026-01336-3

Keywords: deep learning, reproducibility, dynamical systems, riddled basins, fat fractals, neural network training, uncertainty exponent, machine learning theory, Fractal, basin, geometry, limits