Multimodal Artificial Intelligence Model Evaluates Aerobics Routines Across Sensor Streams

A research team has developed an intelligent multimodal evaluation model that combines synchronized video recordings, skeletal pose tracking, wearable inertial measurement units, and musical rhythm patterns to assess competitive aerobics performances. Manual judging in competitive aerobics traditionally depends on visual observation by referees and coaches, leaving evaluations vulnerable to differences in vantage angles, observer experience, and split-second oversights during high-tempo routines. To address these observation challenges, investigators designed an auxiliary evaluation system that captures parallel sensor streams and maps them onto a synchronized action timeline. Across a participant-independent evaluation set of 1,271 routines performed by adult athletes, the model achieved a mean absolute error of 3.18 points on a 100-point scale, demonstrating the viability of integrating biomechanical dynamics with musical timing to assist technical coaching.

Assessing competitive aerobics routines requires tracking rapid combinations of jumps, turns, balance postures, floor supports, and continuous choreography synchronized to musical beats. When relying entirely on standard camera footage, automated vision systems frequently encounter motion blur, perspective distortion, limb occlusions, and background clutter from varying sports hall environments. Conversely, relying solely on skeletal coordinates or wearable accelerometers overlooks either the athlete’s spatial morphology or the auditory cadence dictated by the accompaniment track. The researchers sought to overcome these limitations by establishing a unified pipeline that accepts heterogeneous sensory inputs, aligns them chronologically, and prevents any single dominant signal from overwhelming more subtle technical indicators during distinct phases of athletic movement.

To construct a robust dataset for development and testing, sixty competitive aerobics athletes, comprising 34 women and 26 men aged 18 to 26 years with between one and eight years of specialized training, performed across six defined action categories: aerobic combinations, jumps, turns, balance, support, and routine transitions. Each exercise was executed across standard execution, mild deviation, and obvious deviation levels. The recording setup captured high-definition video at 30 frames per second, mapped 33 human keypoints, collected 100-hertz acceleration and angular velocity readings from five body-worn inertial sensors placed on the torso, wrists, and ankles, and sampled accompanying audio tracks at 44.1 kilohertz. After excluding sequences marked by synchronization errors, sensor cutouts, or extreme physical occlusion, the team curated 8,466 valid motion sequences.

Ground-truth scoring and diagnostic labels were established by an independent expert panel consisting of two national-level aerobics referees, two university coaches with at least eight years of coaching experience, and a physical education professor specializing in aerobics assessment. The panel rated six distinct performance dimensions: technical execution, range of motion, body control, rhythm synchronization, artistic performance, and overall stability. Inter-rater reliability evaluated via two-way random-effects intraclass correlation coefficients yielded high consistency across all six dimensions, ranging from 0.86 for artistic performance to 0.93 for rhythm synchronization. The trimmed mean of expert scores formed the regression targets, while a strict re-scoring protocol resolved cases exhibiting score ranges exceeding eight points, ensuring verified targets for all subsequent machine-learning tasks.

The neural architecture processes the synchronized data through four specialized input branches operating in parallel. Video sequences are cropped around the athlete and processed through a MobileNetV3-Small backbone paired with temporal convolutional layers to map visual form into 128-dimensional representations. Skeletal coordinate sequences enter a three-block spatial-temporal graph convolutional network that analyzes anatomical topology and continuous joint angles. Inertial sensor channels pass through depthwise-separable one-dimensional convolutions and a bidirectional gated recurrent unit, capturing instantaneous velocity spikes, landing impacts, and rotational braking. Simultaneously, accompaniment audio is converted into log-Mel spectrograms and processed via convolutional blocks to extract downbeats, tempo consistency, and execution lead or lag relative to musical cadence.

Following modality-specific encoding, a four-head bidirectional cross-modal attention module allows features from one stream to retrieve relevant information from the others, associating physical joint trajectories directly with acceleration peaks and musical downbeats. Because physical sensors and optical cameras experience fluctuating real-world data quality, the system incorporates a reliability-aware adaptive fusion mechanism. This module dynamically estimates quality vectors, such as keypoint tracking confidence, visible body proportions, and inertial noise ratios, to calculate softmax-normalized fusion weights. Rather than applying rigid weights across the entirety of an exercise, the network adaptively adjusts modal reliance to match specific movements, allowing inertial signals to guide jump impact analysis while skeletal branches dominate static balance assessments.

The network is trained via a multi-task joint loss strategy combining smooth L1 loss across the six sub-item scores, mean squared error on the composite score, focal loss for four-class primary deviation classification, cross-modal contrastive loss, and a batch-level weight balancing penalty. The classification head identifies primary execution errors categorized into amplitude deficiency, posture deviation, control imbalance, or rhythm deviation, defined by the panel’s largest deduction. A temporal attention mechanism identifies critical low-scoring temporal segments across routines. The full deep representation model contains 7.84 million trainable parameters, optimized via the AdamW algorithm across five fixed random seeds to ensure reproducible model training behavior.

In empirical benchmarking against baseline evaluation architectures, the proposed model outperformed conventional CNN-LSTM, spatial-temporal graph networks, I3D-LSTM, and early-fusion transformer models when evaluated under identical participant-independent data splits. Across five fixed-seed runs, the proposed system achieved an average mean absolute error of 3.18 points and a root mean square error of 4.21 points, representing reductions in error of 27.1 percent and 26.0 percent compared to the early-fusion transformer baseline. Coefficient of determination reached 0.946, while Spearman correlation reached 0.963, demonstrating close alignment with composite referee judgments. Furthermore, the model achieved a macro-F1 score of 0.925 in classifying the primary technical deviation across test sequences.

Ablation experiments confirmed that the interaction and adaptive fusion mechanisms were central to these performance gains. When bidirectional cross-modal interaction was eliminated in favor of simple feature concatenation, mean absolute error increased from 3.18 to 4.52 points, while deviation classification macro-F1 dropped to 0.876, representing the largest degradation observed among all tested variants. Replacing adaptive fusion with static, equal weighting across all four streams raised scoring error to 4.08 points. Similarly, removing temporal attention in favor of global average pooling increased error to 3.79 points, confirming that retaining localized temporal indicators is necessary for detecting brief balance faults, landing missteps, and delayed turns.

In robust stress tests altering acquisition conditions, the system demonstrated operational stability under simulated capture perturbations, maintaining a macro-F1 score of 88.3 percent under mixed speed fluctuations, body occlusion, and sensor displacement. An experimental four-week pilot implementation involving 18 athletes and four coaching specialists revealed that candidate error intervals identified by the system assisted review routines, reducing average teacher video review times from 24.6 minutes to 15.3 minutes per recording. However, evaluation under held-out cross-site and cross-device environments increased scoring error to 4.28 and 4.47 points respectively, while high-scoring routines were systematically underestimated. The authors emphasized that the framework serves as an auxiliary diagnostic tool for training analysis rather than an autonomous substitute for certified referee panels.

Analysis of internal fusion weights across the 1,271 test sequences illustrated how the network adjusted its sensory reliance across distinct movement types. For jumping routines, the model assigned its highest average weights to inertial readings (0.39) and skeletal coordinates (0.31), while turning actions similarly emphasized inertial (0.35) and skeletal (0.34) signals to track angular velocity and rotational braking. Conversely, static balance and floor support exercises relied most heavily on skeletal topology, receiving mean weights of 0.41 and 0.43, respectively. Aerobic combinations prioritized optical video (0.31) and rhythm metrics (0.28), whereas routine transitions balanced reliance among video (0.29), skeleton (0.27), and rhythm (0.26). To verify whether the network’s temporal attention corresponded to human instructional priorities, five independent teachers evaluated 300 test sequences without viewing algorithmic outputs. Their identified deduction intervals achieved a temporal intersection-over-union of 0.71 with the model’s high-attention segments and matched the primary error category in 89.0 percent of cases (Fleiss’ kappa of 0.82), confirming the utility of temporal attention as an auxiliary review cue rather than a causal explanation.

Systematic stress testing across unfamiliar domains outlined clear operational boundaries for the framework. When tested on 486 sequences captured in a separate gymnasium featuring camera distances shifted to 7–10 meters, viewing angle changes up to 20 degrees, and lighting variations of 20 to 35 percent, mean absolute error rose to 4.28 points. Similarly, evaluation on 455 sequences recorded with secondary cameras and alternative inertial sensors increased error to 4.47 points and lowered the coefficient of determination to 0.878, though rank-order consistency remained solid at a Spearman correlation of 0.902. Score calibration analyses indicated that while predictions closely tracked expert evaluations across mid-range performances, the model slightly overestimated scores between 50 and 70 points and systematically underestimated elite routines scoring above 90 points. Computational profiling demonstrated practical runtime efficiency, with single-batch inference averaging 38.6 milliseconds per sequence. Nevertheless, because the preliminary four-week pilot lacked a control group or objective markers of technical skill retention, the researchers concluded that autonomous adoption remains premature without preregistered multi-center trials across varied competitive divisions.

Subject of Research: Multimodal artificial intelligence framework for objective aerobics movement evaluation and training diagnosis

Article Title: Optimization of an intelligent evaluation model for aerobics performance using artificial intelligence and multimodal deep representation

Article References: Xue, B., Wang, T., & Zong, H. (2026). Optimization of an intelligent evaluation model for aerobics performance using artificial intelligence and multimodal deep representation. Discover Artificial Intelligence, 6(1), Article 1861. https://doi.org/10.1007/s44163-026-02754-3

Image Credits: AI Generated

DOI: 10.1007/s44163-026-02754-3

Keywords: aerobics evaluation, multimodal deep learning, pose estimation, inertial measurement units, cross-modal attention, adaptive fusion, sports engineering, motion analysis, spatial-temporal graph convolution, feature representation, biomechanics, temporal attention, performance diagnosis, multi-task learning, athletic training, sensor synchronization, computer vision in sports, wearable sensing