Wearable devices such as smartwatches and smartphones have become quiet observers of daily life, logging every step, stair climb, and shift in posture through tiny motion sensors. Turning those raw signals into reliable recognition of what a person is actually doing has long been a stubborn challenge for artificial intelligence, particularly when activities resemble one another closely — think of sitting versus standing, or walking upstairs versus walking downstairs. A new study published in the journal Neural Processing Letters reports a hybrid deep learning architecture that appears to close much of that gap, combining the pattern-grabbing strengths of convolutional networks with the long-range memory of Transformer models, while remaining light enough to run on resource-constrained hardware.
The system, called MLConvTrans — short for Multi-Level Convolutional Transformer — was developed by Thi Hong Vuong of VNU University of Engineering and Technology in Hanoi and the National Institute of Informatics in Tokyo, together with Tung Doan of Hanoi University of Science and Technology and Atsuhiro Takasu of the National Institute of Informatics and SOKENDAI. In benchmark tests spanning six widely used public datasets, the model delivered higher accuracy and F1-scores than a roster of state-of-the-art competitors, all while using fewer trainable parameters, fewer floating-point operations, and shorter inference times than other Transformer-based approaches.
Human activity recognition, or HAR, has expanded rapidly in recent years, finding roles in sports analytics, smart homes, and health monitoring, including fall detection, elderly care, and the tracking of abnormal movements in patients. The basic recipe is straightforward: accelerometers, gyroscopes, and sometimes magnetometers embedded in a wearable or handheld device capture three-axis motion signals, which software then segments into windows and classifies into activity labels such as walking, running, sitting, or jumping. What happens inside that classification stage, however, has been the subject of fierce competition among machine learning approaches.
Early HAR systems relied on classical algorithms such as Support Vector Machines, Decision Trees, and Random Forests. These methods demanded extensive hand-crafted feature engineering — human experts had to decide which statistical properties of the sensor signals mattered — and their accuracy and generalizability suffered as a result. Deep neural networks changed the game by learning features automatically from raw data, stacking layers that capture increasingly abstract patterns. Convolutional neural networks excelled at pulling local features out of multichannel sensor streams, but they struggled to model long-term temporal dependencies across extended sequences. Recurrent architectures, including Long Short-Term Memory networks and bidirectional variants, addressed temporal dynamics but often failed to capture correlations among heterogeneous sensor modalities. Hybrid CNN-LSTM models, which pair spatial feature extraction with temporal modeling, improved matters further yet still left long-range relationships and inter-modal physical correlations underexploited.
Transformers, the architecture family that underpins modern large language models, entered the HAR arena promisingly because their self-attention mechanism naturally captures long-range dependencies across a sequence. But as the authors note, existing Transformer-based HAR systems came with their own baggage. Original Transformer designs struggled to correlate isolated timestamps into cohesive activity representations, often required elaborate pre-processing pipelines to merge signals from multiple sensors — one method needed 273 engineered features extracted before the model could even run — and offered little in the way of efficiency analysis. Variants that combined CNNs with Transformers, such as IF-ConvTransformer and HART, partially alleviated these issues but introduced complexity of their own, whether through physically informed fusion blocks that limit sensor flexibility or through stacks of encoder layers that inflate inference time.
MLConvTrans attacks the problem with a two-stage design. The first component, a Multi-Level Convolutional Network dubbed MLConvNet, handles local feature extraction and multimodal fusion. It operates at two levels. At level one, a Conv Embedding block built from 1D convolutional layers with kernel size three, each followed by max pooling, processes the accelerometer and gyroscope streams separately, extracting sensor-specific local features before concatenating them into a unified embedding. At level two, these projected embeddings flow through three Residual Attention Blocks — convolutional layers with kernel size five, batch normalization, ReLU activation, and skip connections — which aggregate global, contextual information across the fused sensor data. The residual design keeps the network compact while enabling the extraction of complex spatial patterns, and the multilevel structure deliberately balances small kernels for fine-grained local detail with larger kernels for broader receptive fields.
The second component, the TransEncoder, is a stack of two Transformer encoder layers that receives MLConvNet’s output combined with sinusoidal positional encodings. Each encoder layer contains a multi-head self-attention block and a feed-forward multilayer perceptron with Gaussian Error Linear Unit activation. The self-attention mechanism computes queries, keys, and values from learned linear projections of the input, weighting each time step’s contribution to every other time step and thereby modeling long-term dependencies that convolutions alone cannot reach. A final classification block applies average pooling, batch normalization, a multilayer perceptron, and dropout, followed by a softmax that distributes probability across activity classes, trained with a cross-entropy loss.
The experimental campaign was unusually broad. The team evaluated MLConvTrans on UCI-HAR, MotionSense, HAPT, KU-HAR, SHL2018, and PAMAP2 — datasets collected from smartphone- and IMU-based setups covering anywhere from six to eighteen activities, sample rates from 50 to 100 Hz, and participants ranging from 9 to 90 per study. Raw accelerometer and gyroscope signals were channel-wise z-normalized, segmented into 128-timestamp windows, and split 70/10/20 into training, validation, and test sets. Training ran for 200 epochs with a batch size of 64 and the Adam optimizer at a learning rate of 0.0001 on Nvidia Tesla V100 GPUs, with class weighting to counter label imbalance, and every configuration was repeated ten times to produce averaged metrics.
The results were decisive. MLConvTrans achieved average F1-scores of 97.31 percent on UCI-HAR, 98.74 percent on MotionSense, 92.43 percent on HAPT, 95.93 percent on KU-HAR, 80.59 percent on SHL2018, and 96.76 percent on PAMAP2, with corresponding accuracies of 98.26, 98.87, 93.15, 96.82, 81.34, and 97.21 percent. Against CNN-based baselines such as DeepCNN, TCN, and AReNet, the F1-score gains ranged from roughly 4 to 19 percentage points depending on the dataset. Hybrid CNN-LSTM methods trailed by between 1.7 and 12.5 points, and even fellow Transformer approaches — the vanilla Transformer, IF-ConvTransformer, and HART — fell short by 3.1 to 7.8 points. Stratified five-fold cross-validation confirmed the model’s stability, holding above 95 percent accuracy on four of the six datasets.
Confusion-matrix analysis offered a nuanced picture of where errors persist. On UCI-HAR and MotionSense, most misclassifications occurred between standing and sitting — statically similar postures that even humans can confuse from motion data alone. HAPT proved harder because it includes postural transitions such as sit-to-lie and stand-to-lie, brief and unstable movements with few training samples. On SHL2018, a real-world transportation dataset, the model excelled at dynamic activities like walking, running, and cycling, but distinguishing among car, bus, train, and subway rides — where passengers mostly sit still — pushed individual class accuracies down to the 68 to 82 percent range, reflecting the intrinsic ambiguity of the signals rather than any architectural weakness.
Ablation studies confirmed that both components earn their place. Removing the TransEncoder and keeping only MLConvNet cost the model between 2.9 and 12.1 points of accuracy depending on the dataset, while removing MLConvNet and relying on the Transformer alone cost between 2.0 and 5.1 points. The team also tuned the architecture empirically: six convolutional layers paired with two Transformer layers proved the sweet spot, a kernel size of three in level one and five in level two outperformed uniform choices, and four attention heads hit the best accuracy-to-cost balance, with eight heads adding only marginal gains while inflating the parameter count by more than a quarter. Testing additional sensors showed that adding a magnetometer yields only modest improvements, suggesting the accelerometer-gyroscope pair remains the pragmatic choice for most deployments.
Perhaps most consequential for real-world use is the efficiency profile. MLConvTrans contains 1,576,924 trainable parameters and requires about 16.3 million FLOPs per inference — less than every Transformer-based competitor tested, including the vanilla Transformer at 2.01 million parameters and HART at 2.54 million. Its average inference time of 3.52 milliseconds was the lowest in the comparison, beating ConvTransformer, the vanilla Transformer, HART, and IF-ConvTransformer, the latter of which required nearly five milliseconds on the same hardware. That combination of accuracy and speed, the authors argue, makes the architecture genuinely suitable for real-time applications such as fall detection on battery-powered wearables.
The team’s future work aims to shrink the model further, building a lightweight HAR engine that can live comfortably inside smart devices with limited memory, storage, and energy budgets. If MLConvTrans’s benchmark performance holds outside the laboratory, the quiet motion sensors already strapped to millions of wrists and pockets may soon understand human movement considerably better — and understand it quickly enough to matter in the moment.
Subject of Research: Wearable sensor-based human activity recognition using a hybrid multi-level convolutional and Transformer deep learning architecture
Subject of Research: Technology and Engineering
Article Title: MLConvTrans: Multi-Level Convolutional Transformer for Wearable Sensor Based Human Activity Recognition
Article References: Vuong, T. H., Doan, T., & Takasu, A. (2026). MLConvTrans: Multi-Level Convolutional Transformer for Wearable Sensor Based Human Activity Recognition. Neural Processing Letters, 58(3), Article 40. https://doi.org/10.1007/s11063-026-11854-1
Image Credits: AI Generated
DOI: 10.1007/s11063-026-11854-1
Keywords: human activity recognition, wearable sensors, deep learning, convolutional neural network, Transformer, multimodal sensor fusion, self-attention, inertial measurement unit, MLConvTrans, edge computing
Cite Scienmag News
APA MLA Chicago
Blake Davidson. (September 6, 2026). Multi-Level Convolutional Transformer Boosts Wearable Sensor Activity Recognition. Scienmag. https://scienmag.com/multi-level-convolutional-transformer-boosts-wearable-sensor-activity-recognition/
Blake Davidson. “Multi-Level Convolutional Transformer Boosts Wearable Sensor Activity Recognition.” Scienmag, 6 September 2026, https://scienmag.com/multi-level-convolutional-transformer-boosts-wearable-sensor-activity-recognition/. Accessed 6 September 2026.
Blake Davidson. “Multi-Level Convolutional Transformer Boosts Wearable Sensor Activity Recognition.” Scienmag. September 6, 2026. https://scienmag.com/multi-level-convolutional-transformer-boosts-wearable-sensor-activity-recognition/
Copy citation Download RIS
Tags: activity classification accuracybenchmarking activity recognition algorithms on public datasetscombining CNNs and Transformers for sensor dataconvolutional and transformer model integrationconvolutional transformer architecture for activity detectiondeep learning for wearable devicesenhancing activity detection accuracy with multi-level modelshybrid deep learning models for wearable deviceshybrid neural network architectureimproving accuracy in activity recognition with deep learninglightweight AI for embedded hardwarelightweight neural networks for wearable hardwarelong-range memory in activity recognition modelsmotion sensor data analysismotion sensor data analysis for activity classificationmulti-level convolutional transformermulti-level convolutional transformer for sensor datareal-time activity monitoringresource-efficient AI modelsresource-efficient AI models for smartwatchessensor-based human activity recognitionwearable device activity detectionwearable sensor activity recognition
