A new open-access paper in Data Science and Engineering proposes AdamMAD, a modified version of the widely used Adam optimizer that replaces the squared gradient in Adam’s second-moment estimate with the mean absolute deviation of the gradient. The authors, Weiwei Cheng, Bin Wang, Jian Li, Bing Li and Xiaochun Yang, argue that this single substitution produces smoother training-loss curves, fewer oscillations, faster convergence and better handling of sparse gradients, while adding only a small computational overhead. The paper was published on 11 April 2026 in Volume 11, pages 668 to 686 of the journal.
The motivation comes from a well-known weakness in Adam’s design. Adam combines a first-moment estimate, the exponential moving average of the gradient, with a second-moment estimate, the exponential moving average of the square of the gradient. Because squaring magnifies large values, a sudden spike in the gradient inflates the second moment sharply. In the parameter update rule, the step size is the learning rate multiplied by the first moment divided by the square root of the second moment plus a small constant epsilon. When the second moment grows large, the square root in the denominator grows with it, and the effective learning rate becomes overly conservative. The paper formalizes this with a proportionality stating that oscillation scales with the inverse of the learning rate when that rate is too small: updates fail to keep pace with large gradient changes, parameters linger, and when a substantial adjustment finally occurs it arrives abruptly, producing oscillations in the training loss.
To document the problem empirically, the authors ran Adam on the VGG-16 convolutional network over the MNIST and Fashion-MNIST handwritten and fashion-image datasets, and observed significant oscillations and fluctuations in the training loss. They then benchmarked Adam against two popular descendants, AdamW and QHAdam, using a detection threshold of 0.0001 to flag significant loss changes as oscillations, and recorded the maximum amplitude and frequency of those oscillations next to each curve. AdamW, which decouples weight decay from the gradient update, and QHAdam, which blends bias-corrected moments with the raw gradient and squared gradient through discount factors, both showed lower amplitude and frequency than Adam. Yet the authors report that oscillations remained relatively large and frequent for both, because all of these methods still rely on the square of the gradient for the second moment. The paper concludes that the oscillation problem is therefore not fully resolved by either variant.
AdamMAD’s central change is to define the second moment as an exponential moving average of the mean absolute deviation rather than of the squared gradient. The mean absolute deviation is the average of the absolute values of the gradient components, computed as the sum of absolute component values divided by the number of components. The authors prove a theorem that MAD-based updates are smoother than squared-gradient updates. The proof considers a gradient change from time t to t plus one and shows that the change in the squared gradient contains a quadratic cross term that can significantly amplify changes, particularly when the current gradient is large. By contrast, applying the triangle inequality to the absolute values shows that the change in MAD is bounded by the average absolute gradient change, a linear response that avoids the amplification associated with squared terms. The team also retains the square root in the denominator of the update, arguing that it moderates the effective step size, prevents sudden jumps in parameter updates, preserves the adaptive scaling behavior of the original Adam, and keeps AdamMAD compatible with existing frameworks that use learning-rate schedules, weight decay and other Adam-oriented techniques.
The paper backs this design with a theoretical analysis. A bounded-gradient theorem shows that under standard smoothness and boundedness assumptions, with decay parameters between zero and one, positive epsilon and a bounded or constant step size, the gradients, moving averages and update steps generated by AdamMAD remain uniformly bounded, which the authors say keeps the optimizer in a stable regime and avoids divergence. A convergence theorem then establishes that for an unconstrained problem whose objective is bounded, continuously differentiable and has an L-Lipschitz continuous gradient, the expected excess loss above the local optimum falls on the order of a constant divided by t raised to a power beta, where beta lies between zero and one. A corollary comparing Adam and AdamMAD under the same assumptions finds that AdamMAD’s bound constant is smaller than Adam’s, which the authors interpret as a superior convergence bound. A second corollary extends the comparison to AdamW and QHAdam and concludes that AdamMAD achieves faster convergence than both, with its bound constant smaller than theirs.
On the experimental side, the evaluation covers image classification with the VGG series, from VGG-11 to VGG-19, and the DenseNet series, from DenseNet-121 to DenseNet-201, on MNIST, Fashion-MNIST and CIFAR-100. Hyperparameters were chosen through a staged grid-search-like procedure with coarse search, fine search and cross-domain verification, settling on a learning rate of 0.001, decay parameters of 0.9 and 0.999, and epsilon of ten to the minus eight, values that align with commonly used defaults in the literature. All optimizers, including SGD with momentum 0.9, Adam, AdamW, QHAdam, DiffGrad and AggMo, used the same search procedure, a batch size of 128 and 50 training epochs. Experiments ran in PyTorch on an NVIDIA A8000 GPU, and results were averaged over three random seeds.
The image results follow the theoretical predictions. Across the VGG models on MNIST and Fashion-MNIST, the training losses of Adam, AdamW and QHAdam showed significantly greater oscillations in both amplitude and frequency than AdamMAD under the 0.0001 threshold, and AdamMAD’s loss curve consistently stayed below the other three. On CIFAR-100, where all optimizers oscillated less, AdamMAD still matched or beat the baselines, with amplitude and frequency values significantly lower than or equal to the others and its curve remaining below or partially overlapping the competitors. Test accuracy curves on CIFAR-100 with the VGG architectures showed smoother accuracy improvement and faster convergence for AdamMAD. On the DenseNet models trained on MNIST, AdamMAD’s loss curve again remained below or partially overlapped the baselines. Epoch-time tables for all six optimizers on the three image datasets indicate that the overhead of computing MAD is minimal, and the authors report that AdamMAD reached better performance in fewer epochs while remaining competitive with DiffGrad and AggMo.
To test generality beyond vision, the team ran additional experiments on text, tabular and time-series tasks: AG News and R52 news classification with a single-layer BiLSTM using embedding and hidden dimensions of 128, the Adult income binary-classification dataset with a three-layer multilayer perceptron of hidden dimension 128, and ETTh1 hourly sensor readings with a two-layer LSTM forecasting the next 24 hours from 96-hour input windows. These runs used a batch size of 64, a learning rate of 0.001 for adaptive optimizers or 0.01 for SGD, 15 epochs with early stopping patience of 5, cross-entropy loss for classification and mean squared error for forecasting, with three random seeds and results reported as mean plus or minus standard deviation. Across the text and tabular benchmarks, AdamMAD improved validation accuracy by roughly 0.1 to 0.5 percent over AdamW and QHAdam with smaller variance across seeds, and on ETTh1 it reduced mean absolute error by about 1.3 percent and root mean squared error by 1.1 percent relative to AdamW while needing fewer epochs to reach 95 percent of its best performance. Paired t-tests over the three seeds marked improvements with p below 0.05 as statistically significant in the results tables.
The discussion section adds a convergence-efficiency claim: in CIFAR-10 and ETTh1 experiments, AdamMAD reached 95 percent of its final performance roughly 15 to 25 percent faster than Adam and AdamW. The authors conclude that AdamMAD improves optimization stability, convergence speed and generalization robustness across diverse domains, and position it as a strong and versatile alternative to existing adaptive optimizers. They also flag future work, including validation on more tasks and datasets, integration with techniques such as learning-rate scheduling and adaptive gradient clipping, further theoretical analysis and research on automatic hyperparameter tuning. The datasets used, including MNIST, Fashion-MNIST and CIFAR-100, are publicly downloadable, and the article is open access.
For practitioners, the takeaway is that a modest, well-motivated change to the second-moment estimator may deliver measurable gains where Adam-family optimizers struggle most: loss spikes, noisy or sparse gradients and slow convergence. The evidence presented spans image, text, tabular and time-series domains, multiple architectures and statistical significance testing, though the reported accuracy gains on text and tabular tasks are modest in absolute terms and the theoretical bounds rely on assumptions such as Lipschitz-continuous gradients and bounded objectives that may not hold exactly in every deep-learning setting. Readers can consult the full paper, titled AdamMAD: Improved Second-Moment of Adam for Enhancing Stability and Convergence, for the complete derivations, tables and figures.
Subject of Research: A modified Adam optimizer that uses mean absolute deviation instead of squared gradients in its second-moment estimate to improve training stability and convergence.
Article Title: AdamMAD: Improved Second-Moment of Adam for Enhancing Stability and Convergence
Article References: Cheng, W., Wang, B., Li, J., Li, B., & Yang, X. (2026). AdamMAD: Improved Second-Moment of Adam for Enhancing Stability and Convergence. Data Science and Engineering, 11(3), 668-686. https://doi.org/10.1007/s41019-025-00341-7
Image Credits: AI Generated
DOI: 10.1007/s41019-025-00341-7
Keywords: AdamMAD, Adam optimizer, deep learning, optimization, second moment, mean absolute deviation, training stability, convergence, neural networks, machine learning, gradient descent, VGG
News Source: Blake Davidson. (November 5, 2026). New AdamMAD Optimizer Swaps Squared Gradients for Mean Absolute Deviation to Cut Training Oscillations. Scienmag.
