New Federated Learning Aggregation Rules Keep AI Training Stable When Client Data Diverge

Federated learning lets many devices train a shared machine learning model without ever shipping their raw data to a central server, a property that makes the approach attractive for privacy-sensitive applications. Yet the same design creates a stubborn technical problem: when the data held by different clients differ sharply from one another, a situation researchers call non-IID data, the locally trained models drift apart and the merged global model suffers. A new study published in Cluster Computing proposes two lightweight aggregation algorithms, dubbed CROG and CLIG, that counter this drift by weighting each client’s contribution according to how far its update strays from a trusted reference point.

The work, authored by Jang Hoon Bae, Byoungjun Park, Pedro Porto Buarque de Gusmão, Minchul Kim and Minhoe Kim, appeared online on 22 October 2026 as volume 29, article number 875 of the journal. Its central insight is that the degree of local client drift can be quantified with gradient distances, and that these distances can then be folded directly into the server’s aggregation strategy. Clients whose updates deviate strongly receive smaller weights, while clients whose updates stay close to the reference receive larger ones, damping the influence of the most divergent participants.

In a typical federated learning round, a server selects a subset of clients, sends each a copy of the current global model, and lets every participant train locally on its own dataset before uploading gradients. The standard FedAvg algorithm then averages these updates in proportion to the number of data points each client holds. The authors point out that this data-volume weighting ignores heterogeneity entirely: a client with many examples from a very different distribution can dominate the aggregate and pull the global model away from a good compromise.

CROG, short for Cross-ROunds Gradients, takes the previous round’s global model as its reference. It computes the L2 norm distance between that global model and each client’s current local model, then assigns aggregation weights inversely proportional to these distances. The intuition borrows from proximal methods such as FedProx: a local model that has wandered too far from the global model of the previous round is likely to harm learning, so its influence on the merged update is reduced. The weights are normalized so they sum to one across participating clients.

CLIG, or Cross-CLIents Gradients, instead uses a spatial reference computed within the same round. The server first forms the FedAvg-weighted aggregate of the participating clients’ gradients, then measures each individual client’s L2 distance from that aggregate. Clients close to the collective update gain weight, while outliers lose it, a mechanism the authors describe as conceptually similar to majority voting. Both methods rely on the magnitude of Euclidean gradient distance rather than directional measures such as cosine similarity, and neither requires proxy data, extra client-side computation, or additional communication, which the authors argue distinguishes them from earlier adaptive aggregation schemes that depend on IID proxy samples or heavier client operations.

The paper backs the design with theory. The authors prove that the inverse-distance weights uniquely minimize the maximum weighted deviation any single client can contribute to the server update, a minimax property that equalizes every client’s weighted-deviation contribution. Under standard smoothness assumptions and an inexact-descent condition, they derive a convergence bound in which the optimization error decomposes into the usual term shrinking with the number of rounds plus a heterogeneity term controlled by the worst-case deviation bound. Highly deviated clients therefore cannot dominate the update, and the convergence penalty stays bounded.

The empirical evaluation spans three datasets, Fashion-MNIST with a multilayer perceptron and CIFAR-10 and CIFAR-100 with convolutional networks, using Dirichlet-distributed label splits to simulate varying degrees of non-IIDness. Against baselines including FedAvg, FedNova, FedProx, SCAFFOLD, FedDyn, MOON, FedAdp and FedLAW, the proposed algorithms outperformed all baselines in nearly every tested configuration, with the single exception of Fashion-MNIST at the most extreme heterogeneity setting reported in the main table. Under severe heterogeneity, several baselines dropped below 30 percent accuracy, while the proposed methods remained robust. FedLAW, which consumes roughly 200 IID proxy samples drawn from the test set, was treated not as a fair comparator but as an upper bound achievable when extra IID data is accessible.

The advantage widened as conditions grew harsher. When the participation ratio dropped from full participation down to 0.25, the performance gap over baselines increased, and FedAdp’s degradation at low participation was attributed to outdated momentum in its smoothed angle statistic. With 100 clients and a participation ratio of 0.1 at a Dirichlet concentration of 0.05, either CROG or CLIG was the best method while most baselines suffered severe degradation. In the harshest settings without hyperparameter tuning, FedLAW failed in 5 out of 5 runs, whereas the proposed methods trained stably. Measured by communication rounds needed to reach a target accuracy, the new algorithms trained up to 1.5 times faster than FedAvg in challenging environments. All main results average five runs with different random seeds, reporting test accuracy over the final ten of 200 rounds.

Supporting analyses show the mechanisms working as intended. Weighted gradient distances, in both mean and standard deviation, decreased relative to FedAvg for both algorithms, indicating effective smoothing of heterogeneous updates. Loss-landscape visualizations over aggregation weights placed CROG and CLIG closer to brute-force optimal weights than FedAvg, with average Euclidean distances to the optimum of 0.23 and 0.35 versus 0.41 for FedAvg over 200 rounds. The methods also produced a more balanced effective class distribution among participating clients, lowering the standard deviation of the effective data ratio across rounds.

The authors are careful about limits. CROG and CLIG provide no formal differential privacy guarantees, no privacy-oriented evaluation was conducted, and the interaction with differentially private federated learning mechanisms remains unexplored. CROG outperformed CLIG in the most extreme environments, plausibly because CLIG’s majority-voting character suffers more when few clients participate. Future work, the paper notes, should address privacy-preserving extensions, tasks beyond visual classification, and practical conditions such as client dropouts, noisy updates, device-dependent delays and varying local epoch schedules. Even so, the results suggest that a simple distance-aware weighting rule, requiring no proxy data and no client modification, can materially stabilize federated training where data are most uneven.

Subject of Research: Adaptive aggregation algorithms that improve federated learning robustness under highly heterogeneous, non-IID client data.

Article Title: Robust federated learning aggregation for highly heterogeneous data

Article References: Bae, J. H., Park, B., Gusmão, P. P. B. D., Kim, M., & Kim, M. (2026). Robust federated learning aggregation for highly heterogeneous data. Cluster Computing, 29(15), Article 875. https://doi.org/10.1007/s10586-026-06716-3

Image Credits: AI Generated

DOI: 10.1007/s10586-026-06716-3

Keywords: federated learning, non-IID data, adaptive aggregation, gradient distance, machine learning, distributed training, client drift, CROG, CLIG, convergence, Robust, federated