Smart sensors that never forget: three learning paradigms collide in activity recognition

Every time you wave a smartwatch over your wrist, ask a smart speaker to dim the lights, or let a fall-detection sensor watch over an elderly relative, a machine learning model is quietly deciding what your body is doing. A new mini-review published in Applied Intelligence argues that the technology behind those split-second judgments is hitting a wall, and that the way past it requires three branches of artificial intelligence to stop being developed in isolation. The paper, by André L. S. Conde, Rodolfo I. Meneguette, Robson E. De Grande and Caetano M. Ranieri, maps out what happens when multimodal learning, federated learning and continual learning are forced to work at the same time on human activity recognition, and why almost nobody has managed to combine all three.

Human activity recognition, or HAR, is the task of automatically classifying what a person is doing, from walking and running to cooking or doing the laundry. It sounds simple, but real-world deployment is notoriously fragile. Sensors shift position on the body, users behave in ways the training data never anticipated, and complex activities are chains of sub-actions that a single accelerometer cannot disambiguate. The review’s central claim is that in realistic deployments, three problems arrive simultaneously: the data arrive in many different forms, the data are scattered across private devices that cannot be pooled, and the set of activities to recognize keeps growing. Solving any one of these in the laboratory while ignoring the other two, the authors contend, is why so many promising HAR systems fall apart outside controlled environments.

The first pillar is multimodal learning, which fuses heterogeneous signals such as accelerometer and gyroscope readings, audio, video and ambient sensor streams into a single model. The review organizes fusion strategies into three families. Early fusion merges raw features into a common representation before learning begins, preserving inter-modal correlations but demanding that all sensors be available and synchronized. Late fusion lets each modality run independently and combines only the final decisions, which makes the system resilient when a camera drops out but forfeits the chance to exploit cross-sensor complementarities. Intermediate fusion, the trend the authors identify in recent work, processes each modality through specialized encoders and merges their representations in deeper layers using attention or graph-based mechanisms, capturing rich dependencies at the price of higher computational cost.

The second pillar, federated learning, tackles privacy. Activity data are intimate, and they live on the very devices that collect them. Federated learning lets a shared global model be trained by sending only parameter updates to an aggregation server, never the raw data. The canonical algorithm, FedAvg, computes a weighted average of local models, while variants such as FedProx add regularization to stabilize training when clients hold wildly different data distributions. But the review is blunt about the paradigm’s soft spots: poisoning attacks can corrupt the global model, reconstruction attacks can leak sensitive information from the updates themselves, and non-IID data, where each user’s activity habits differ, can push the global model toward overfitting a dominant subset of clients.

The third pillar, continual learning, addresses the fact that life does not come with a fixed label set. A HAR system deployed in a smart home will eventually need to recognize activities nobody thought to include at launch. The review adopts the standard taxonomy of task-incremental, domain-incremental and class-incremental scenarios, and singles out class-incremental learning as the hardest: the model must identify both which activity is being performed and that it is a new class, all without a task identifier. The enemy here is catastrophic forgetting, the well-documented tendency of neural networks to overwrite old knowledge when trained on new tasks. The authors survey four families of countermeasures: replay of stored or generatively synthesized examples, regularization methods such as Elastic Weight Consolidation that penalize changes to parameters deemed important, knowledge distillation in which an older model teaches a newer one, and dynamic architectures that physically grow to accommodate new classes.

Each of these countermeasures, however, collides with the other two paradigms, and it is here that the review makes its sharpest contribution. Replay conflicts directly with federated learning’s founding promise: you cannot revisit past client data you were never allowed to collect. Generative replay, in which a compact generator synthesizes samples of old classes instead of storing them, as in the GenFCIL framework, partially dissolves the tension, but the generator itself becomes a new privacy-sensitive artifact whose fidelity must be traded against its size. Personalization, which helps individual devices cope with their own data quirks, risks fragmenting the global model so thoroughly that a new activity learned by one household never propagates to the rest of the federation, defeating the purpose of collaboration. And dynamic architectures that expand to learn new classes break the assumption, implicit in standard aggregation schemes like FedAvg, that all clients share the same network structure.

The review also documents a phenomenon it calls modality-specific client drift. In a federated HAR deployment, one device might carry only a wearable while another also hears ambient sensors, and that configuration can change over time as devices join, leave or lose connectivity. A client missing an entire modality is not merely underweighted; it operates on a fundamentally different input space, so its parameter updates are not directly comparable with those of multimodal clients. When that same client also sees a skewed, long-tailed activity distribution, where walking and sitting dominate while safety-critical events like falls are rare, its local model can drift toward a minimum misaligned with the global objective. The authors note that this divergence compounds across successive incremental updates rather than resetting between rounds, and that standard federated divergence metrics do not yet measure it.

On the solutions side, the review highlights recent federated multimodal frameworks as signposts. FedFusion compresses client representations into low-dimensional subspaces via singular value decomposition to cut communication costs. FedMVD treats each modality combination as a multi-view domain and aligns them semantically through a global logit alignment mechanism. FedMIR goes further, reconstructing the representation of a missing modality from the ones that are present using contrastive inverse generation, then routing the completed representations through a mixture-of-experts backbone that adapts to distribution drift on the fly. And outside the HAR literature entirely, the authors point to evidence that federated fine-tuning of lightweight adapter modules, with the shared backbone frozen, can reduce per-round communication by more than an order of magnitude, a result they consider directly transferable to sensor-based HAR architectures.

The paper closes with a diagnosis that may sting the field’s benchmark culture: the infrastructure for measuring progress barely exists. Widely used datasets such as PAMAP2, Opportunity and UTD-MHAD were never designed with federated client structures, temporal task boundaries or modality dropout in mind, so researchers improvise partitioning schemes that make cross-study comparisons nearly meaningless. Federated simulations are routinely built on data from fewer than twenty subjects, a client population far smaller than any real smart-environment deployment. Task ordering, a known confound in continual learning, is rarely reported in enough detail to replicate, and metrics for forgetting, communication cost and modality robustness are seldom reported together. The authors call for benchmark suites that treat federated partitioning, task sequencing and modality availability as configurable, reproducible protocol parameters, alongside standardized reporting templates. Until then, they argue, the promise of activity recognition systems that are simultaneously accurate, private and adaptable, in hospitals, homes and assistive technologies, will remain a laboratory achievement rather than a deployed reality.

Subject of Research: Intersection of multimodal, federated and continual learning for human activity recognition

Article Title: Multimodal continual federated learning for human activity recognition: Trends and challenges

Article References: Conde, A. L. S., Meneguette, R. I., De Grande, R. E., & Ranieri, C. M. (2026). Multimodal continual federated learning for human activity recognition: Trends and challenges. Applied Intelligence, 56(16), Article 515. https://doi.org/10.1007/s10489-026-07562-x

Image Credits: AI Generated

DOI: 10.1007/s10489-026-07562-x

Keywords: human activity recognition, federated learning, continual learning, multimodal learning, catastrophic forgetting, privacy, wearable sensors, smart environments, class-incremental learning, knowledge distillation, benchmark datasets, Applied Intelligence