Checkpoint ensembles from ordinary training runs rival specialized noisy-label detectors

A strikingly simple recipe for finding mislabeled training data has been shown to match, and in some cases beat, sophisticated noise-detection methods that rely on foundation models or heavily modified training procedures. The approach, described in a study published in Neural Computing and Applications, requires nothing more than a standard training run: it passively collects model checkpoints around the early stopping point and uses their combined predictions to flag samples whose labels are probably wrong. The authors, Lea Hergert and Márk Jelasity, report that this checkpoint-ensemble strategy is competitive on both noisy-label detection and noise-filtered training across a range of synthetic and human-annotated noise benchmarks, a result they describe as somewhat surprising given that state-of-the-art methods use specialized techniques such as optimized thresholds, external foundation models, or dedicated training procedures.

The problem the researchers tackled is a familiar one in machine learning. Supervised classifiers are trained on datasets in which some labels are simply wrong, whether because of human annotation error, automated crawling mistakes, or deliberate corruption. If those corrupted labels can be identified, annotators can focus their corrections on the problematic samples, or the samples can be removed before training to improve the generalization of the final model. A naive way to find them is to train a well-generalizing model and mark every sample it mispredicts as noisy. This idea has long been used as a baseline, for example in methods such as INCV, but it has generally not been considered competitive with state-of-the-art detectors.

What makes a fresh look worthwhile is a body of theoretical and empirical work showing that gradient descent fits clean samples first and memorizes noisy ones only later in training. In the early phases, the optimizer largely ignores mislabeled examples; memorization sets in during later stages and eventually leads to overfitting, even when large amounts of label noise are present. This clean-priority learning phenomenon suggests that early-stopped models should, in principle, be able to distinguish clean from noisy samples. The new study asks whether, if the details are handled correctly, this natural signal can be turned into a detector that rivals specialized methods, without modifying the training algorithm or relying on external pretrained models.

The answer, according to the authors’ empirical investigation, is yes, provided three details are respected. First, the early-stopped model must be applied to its own training set rather than to an independent dataset, which is methodologically acceptable because the goal is to clean the training set itself. Second, instead of relying on a single model, the method uses an ensemble of checkpoints collected close to the early stopping point, which reduces the variance of the predictions and improves detection accuracy. Third, the learning rate scheduler matters far more than one might expect, because it shapes when generalization peaks and when memorization begins.

The study introduces two detection methods built on this foundation. The first, called ES-Prediction, classifies a sample as noisy if the model checkpoint predicts its assigned label incorrectly. The second, ES-BGSS, is more subtle and is inspired by theoretical results from Liu and colleagues on clean-priority learning. It exploits the gradient of the loss with respect to the bias parameters of the feature layer, the layer just before the logit layer. The sum of these partial derivatives, termed the bias gradient sum or BGS, tends to be negative for clean samples and positive for noisy ones, giving a natural, theoretically motivated decision threshold of zero that requires no tuning.

The intuition behind the bias gradient sum can be traced through the algebra of cross-entropy loss. For a clean sample, the assigned class equals the true class, and the coefficient multiplying the true-class contribution in the gradient is non-positive, since the model’s predicted probability for that class is subtracted from one. For a mislabeled sample, the true class is not the assigned one, and the corresponding coefficient is non-negative. The authors find empirically that the full bias gradient sum correlates reliably with the true-class component, so its sign separates clean from noisy samples. On CIFAR-10 setups, the distributions of BGS values for clean and noisy samples remain concentrated on opposite sides of zero with limited overlap, and because the rule depends only on the sign, isolated extreme values have little influence on the decision. On Tiny ImageNet, however, the overlap between the two distributions is substantially larger and the margin around zero noticeably smaller, suggesting that the effectiveness of the zero-threshold assumption depends on how distinctly clean-priority learning separates the gradient statistics.

The choice of learning rate scheduler turned out to be decisive. In experiments on CIFAR-100 with 20 percent uniform label noise, training a ResNet-18 network with mini-batch stochastic gradient descent and a batch size of 128, the researchers compared multistep, linear, and restarted cosine schedulers. With a well-calibrated multistep scheduler, the early stopping point, the best noise-detection performance, and the best task accuracy all coincide, producing a clear turning point in the training dynamics. With continuous schedulers such as the linear or cosine schedule, these critical points can drift far apart. In the linear case, the best-generalizing model can be useless for noise detection because by then the training data has already been memorized, a consequence of the double-descent phenomenon and so-called benign overfitting. The recall of noise detection dropped dramatically after the early stopping point under all schedulers, as memorization progressively taught the model to reproduce even the wrong labels.

Two ensemble-selection strategies were explored. The first, top-k task loss, picks the k checkpoints with the best validation loss while ignoring everything after the early stopping tolerance window, guarding against the second descent. The second, called Plateau, takes k checkpoints from the epochs just before the multistep scheduler first drops the learning rate. The rationale is that these epochs, spent at the initial learning rate, are unlikely to become trapped in narrow valleys of the loss landscape, so if the initial learning rate is set so that memorization does not occur, the collective decision of such models should be well suited to noise detection. Because checkpoints can be aggregated online during training, storing only a running partial aggregate, the extra storage cost is negligible and the original training algorithm needs no modification. In hyperparameter experiments, ensembles of ten checkpoints proved a robust choice for the top-k strategy, majority voting worked best for ES-BGSS and Plateau, and mean aggregation was slightly better for ES-Prediction with top-k ensembles.

In a comparison against established baselines, including Area under Margin, Confident Learning, and SimiFeat, the last of which uses a CLIP-pretrained ViT-B/32 feature extractor, the new methods held their own. ES-BGSS was competitive across the board, in many cases beating the foundation-model-based SimiFeat variants and winning aggregated categories such as the F1 score, with notably high precision, meaning few clean samples were wrongly flagged. ES-Prediction, despite its striking simplicity, beat specialized methods in several settings. Plateau achieved the best recall of all methods tested, correctly catching most noisy samples, though at the cost of lower precision. The evaluation covered uniform, flip, and instance-based noise models on CIFAR-10, CIFAR-100, and Tiny ImageNet, with the full experimental campaign consuming roughly 1000 GPU hours on NVIDIA RTX 3070 and A6000 GPUs.

The authors also tested their detectors as filters before ordinary training, using the human-annotated CIFAR-10N and CIFAR-100N benchmarks, which contain real-world annotation noise of 18.12 percent and 40.2 percent respectively, and the larger Clothing100k and Food-101N datasets. On these tasks, Plateau filtering reached accuracy comparable to SimiFeat on CIFAR-10N and superior on the harder CIFAR-100N, despite using no external knowledge. Notably, on Clothing100k the plain noisy training run without any filtering, relying only on well-calibrated hyperparameters and early stopping, outperformed all filtering results reported in related work, and Plateau improved on it further. The study also cautions that filtering can sometimes hurt: aggressive removal of samples can discard clean but hard examples and shrink the training data, and implicit noise suppression from early stopping already does much of the work. The authors note limitations, including the strong dependence on the training procedure and scheduler, and the fact that the zero threshold of BGSS, while theoretically motivated, could be tuned further for specific datasets. Code is publicly available, and the methods rely exclusively on standard public datasets.

Subject of Research: Detecting noisy labels in training datasets using ensembles of model checkpoints from early-stopped training runs

Article Title: Detecting noisy labels using checkpoint ensembles

Article References: Hergert, L., & Jelasity, M. (2026). Detecting noisy labels using checkpoint ensembles. Neural Computing and Applications, 38(20), Article 846. https://doi.org/10.1007/s00521-026-12575-1

Image Credits: AI Generated

DOI: 10.1007/s00521-026-12575-1

Keywords: noisy labels, label noise detection, checkpoint ensembles, early stopping, clean-priority learning, bias gradient sum, learning rate scheduler, deep learning, CIFAR-10N, noise-filtered training, machine learning, Detecting