AI reads smiles from video alone, but a simpler model matches it

A camera pointed at the mouth can now track the strength of a smile without a single electrode touching the face, according to a proof-of-concept study published in Frontiers in Bioengineering and Biotechnology. Researchers at Saarland University and the Center for Digital Neurotechnologies Saar in Germany built a deep learning framework called ViMG, short for Video Myography, that predicts the window-averaged amplitude of the surface electromyography envelope of the zygomaticus major, the muscle that pulls the mouth corners upward during smiling, directly from facial video. The system achieved a mean Pearson correlation of 0.661 between its predictions and the reference envelope across five subject-independent test folds, with fold-level values ranging from 0.536 to 0.872.

Conventional facial sEMG relies on pairs of contact electrodes placed on the skin over the target muscle. The authors argue that this physical requirement creates problems that go beyond inconvenience. Electrodes can restrict natural facial movement, cause discomfort during prolonged use, and make participants aware that they are being monitored, which may alter the very expressions a study is trying to capture. In developmental and cognitive research, particularly with children or individuals with neurodevelopmental conditions such as autism spectrum disorder, tactile sensors can induce sensory distress and produce behavioral artifacts that obscure the cognitive states under investigation. Smaller pediatric faces also shorten the distances between electrodes and adjacent muscles, increasing the chance that an electrode pair registers volume-conducted activity from neighboring mimetic muscles rather than the target muscle alone.

ViMG does not attempt to reconstruct the raw electrical waveform. Instead, it solves a deliberately simplified problem: for every 0.7-second window of video, corresponding to 56 consecutive frames, the model outputs a single scalar value that estimates the mean amplitude of the processed root-mean-square sEMG envelope over that window. The reference signal was recorded bilaterally from the left and right zygomaticus major using 4-millimeter Ag/AgCl passive electrodes in a bipolar configuration, sampled at 4,800 Hz, notch filtered at 50 Hz, band-pass filtered between 60 Hz and 1,000 Hz, and converted into an RMS envelope with a 1-second moving window before being resampled to 80 Hz to match the video frame rate. Because the envelope was normalized per recording, the model estimates a dimensionless, trial-relative proportion of that trial’s amplitude range rather than an absolute value in microvolts.

The architecture combines a 3D U-Net with a Long Short-Term Memory network. The encoder-decoder structure of the U-Net preserves spatial detail from grayscale crops of the perioral region, the area surrounding the mouth, while the LSTM was intended to model temporal dependencies in facial muscle activation. In total, the model contains 23,368,994 trainable parameters. Video was captured at 160 frames per second with 3-millisecond exposure under standardized laboratory lighting averaging 1,245 lux, then down-sampled to an effective 80 fps. Four MediaPipe facial landmarks defined the perioral crop, which was converted to grayscale and resized to 70 percent of its stored dimensions. The zygomaticus major electrodes sat on the upper cheek, outside the crop, so the model never saw the electrode sites themselves.

The study involved 23 young adults drawn from an initial cohort of 30; seven were excluded before any model training for technical or procedural reasons, including recording failures, frame drops, spectacle reflections corrupting the perioral region, and one participant whose movements did not follow the instructed exercises. Participants watched video stimuli displaying facial yoga expressions and held each for 10 seconds. The sequence included puckering the lips, drawing the eyebrows together, smiling, raising the eyebrows, and repeating the lip pucker, with eyebrow tasks serving as negative controls and 30-second neutral-expression baseline periods interspersed. Head fixation kept participants still, since camera-based measurement is sensitive to motion artifacts, and all exclusion decisions were made during data screening, independent of model performance.

Evaluation used subject-level five-fold cross-validation, meaning each participant was held out for testing exactly once, and learning-rate scheduling, early stopping, and checkpoint selection were driven exclusively by a validation set of four subjects per fold. At the subject level, correlations were Fisher z-transformed for a one-sample test against a null of zero correlation, and a large effect size was observed for 18 of the 23 participants. However, the variance across subjects was substantial, with a subject-level standard deviation of 0.412, and driven partly by a few participants with negative correlations. In two of those subjects, spontaneous facial movements created sharp sEMG envelope peaks without clear visual correlates, and one subject was recorded with an anomalous frame count that may reflect an acquisition error.

Perhaps the most striking finding is what the ablation experiments revealed. When the researchers removed the LSTM and replaced it with a simple position-wise projection, performance did not drop; the mean fold-level correlation was actually 0.677 without the recurrent layer, against 0.661 with it, and error metrics were lower. Replacing the custom weighted loss function with an unweighted mean squared error likewise yielded a mean correlation of 0.673 with lower error. Across all variants, mean correlations fell between 0.661 and 0.677, a range well within the fold-level standard deviations of 0.09 to 0.13. The authors conclude they find no evidence that either the recurrent layer or the loss weighting contributes to performance on this task.

A learned landmark-based baseline, using only 22 kinematic features per frame derived from four facial landmarks and a single-layer GRU with 19,009 trainable parameters, roughly three orders of magnitude fewer than the full model, achieved a mean subject-level correlation of 0.670 with lower MAE and RMSE than the full model. It was also more consistent across subjects, with no per-subject correlation below zero, whereas the full model had four subjects below zero. The authors interpret this as showing that much of the recoverable signal is explained by coarse perioral kinematics, and that the present results do not demonstrate an advantage for pixel-level spatiotemporal modeling on this task. Predictions also scaled with the reference at a mean slope of 0.42, well below unity, indicating that ViMG recovers the relative time course of the envelope while compressing its amplitude range.

Gradient-based saliency maps, generated by computing the gradient of the model’s prediction with respect to each input pixel, showed that predictions were informed by anatomically plausible perioral regions, with high-intensity attribution concentrated around the edges of the mouth and the nasolabial folds, the smile lines corresponding to the known area of influence of the zygomaticus major. Attribution was notably lower in lateral regions corresponding to the buccinator muscle belly. The authors caution that a saliency map indicates which pixels influence the output but does not establish that the model has isolated a specific muscle or its contraction direction, and they observed high attribution along map borders for a subset of subjects, possibly a padding artifact from zero-padded convolutional layers.

Task-specific analysis complicated the specificity story. The target Smile task achieved a median correlation clearly separated from the Fish Mouth controls, which showed negative median correlations consistent with mechanical coupling at the modiolus, the shared insertion where the zygomaticus major and orbicularis oris converge. But the eyebrow-raising control task, intended as a negative control, yielded a median correlation exceeding that of the Smile task. Post-hoc inspection revealed non-negligible zygomaticus major activity in the reference channel during that task, with plausible explanations including genuine co-activation, mechanically coupled deformation propagating from the upper face, electrical crosstalk, or visual features covarying with the reference signal. The authors report this as an observation that qualifies the interpretation of the condition as a negative control rather than evidence of a specific physiological mechanism, and no formal statistical comparison between task conditions was performed.

The authors are explicit that ViMG does not measure electrical muscle activity. Surface electromyography records the electrical manifestation of motor unit recruitment, whereas ViMG infers an sEMG-derived amplitude measure from its mechanical consequences, namely visible perioral skin deformation. These quantities are physiologically related but not identical. Because the approach is optical rather than electrical, it is not affected by the volume conduction that gives rise to electrical crosstalk in traditional sEMG, but it exchanges electrical crosstalk for mechanical coupling effects rather than eliminating signal mixing altogether, since adjacent facial muscles are mechanically coupled and deformation produced at one site can propagate to another.

Temporal resolution is another constraint. The model produces one estimate per 0.7 seconds, derived from an envelope smoothed over 1 second, giving an effective temporal resolution of approximately 1.7 seconds. This is appropriate for tracking the aggregate intensity of macro-expressions, which unfold over durations exceeding 500 milliseconds, but substantially coarser than conventional sEMG and unsuitable for micro-expressions occurring in under 200 milliseconds, which are not resolvable in principle. The authors also note a fundamental temporal mismatch: sEMG can detect pre-kinematic muscle activation that precedes visible facial movement, while video-based prediction is constrained to observable kinematic changes, so early-phase muscle recruitment below the threshold of detectable movement cannot be inferred from visual data alone.

The cohort of 23 young adults limits generalizability across age groups and facial anatomies, and the head fixation device and standardized lighting represent a controlled laboratory setting that differs from unconstrained natural environments. Dense facial hair could in principle obscure the luminance changes on which the method relies, though no systematic relationship was observed in this cohort. The fixed perioral crop dimensions do not account for variations in head size or camera distance. Future work outlined by the team includes expanding validation to diverse populations, extending the framework to multiple facial muscles, replacing the fixed crop with a subject-specific dynamic crop, validating against spontaneous expressions in natural settings, removing the head fixation requirement, and integrating depth or thermal imaging. The authors position ViMG not as a replacement for clinical electromyography but as a proof of concept for contactless physiological assessment within controlled environments, relevant to cognitive and developmental research where removing contact sensors helps preserve ecological validity, and potentially applicable to populations such as infants or individuals with neurodevelopmental disorders for whom sensor placement is itself a confound.

Subject of Research: Contactless estimation of facial sEMG envelope amplitude from video using deep learning

Article Title: ViMG: a contactless deep learning framework for estimating sEMG amplitude envelopes from facial videos

Article References: Nizamani, F. A., Bhamborae, M. J., Bonnaire, S., Corona-Strauss, F. I., & Strauss, D. J. (2026). ViMG: a contactless deep learning framework for estimating sEMG amplitude envelopes from facial videos. Frontiers in Bioengineering and Biotechnology, 14, Article 1945313. https://doi.org/10.3389/fbioe.2026.1945313

Image Credits: AI Generated

DOI: 10.3389/fbioe.2026.1945313

Keywords: surface electromyography, deep learning, facial expression, zygomaticus major, contactless sensing, video myography, 3D U-Net, LSTM, affective computing, perioral kinematics, saliency mapping, cross-validation