Compact audio-visual AI reads sentiment from live video with 84.89% accuracy

A team of researchers in Thailand and Japan has built a compact neural network that reads a person’s sentiment from live video streams by combining facial expressions and speech, achieving 84.89 percent accuracy on a three-class task while using roughly ten times fewer parameters than comparable audio-visual systems. The work, published in Frontiers in Artificial Intelligence, also benchmarks nine large language models on the same problem and finds they fall well short of the specialized model when audio and visual information must be interpreted together.

The framework, developed by Atitaya Yakaew and Matthew N. Dailey at the Asian Institute of Technology together with Teeradaj Racharak and Watanee Jearanaiwongkul at Tohoku University, classifies affective states as negative, neutral, or positive. It was designed from the outset for low-latency operation on continuous video, the kind of feed produced by cameras already installed in retail and service environments. The authors argue that such streams are a largely under-used resource for understanding customer reactions, since they capture facial expressions and speech simultaneously without requiring people to fill in questionnaires or take part in focus groups.

Architecturally, the system consists of four lightweight deep neural networks. A face and mouth detection stage uses a histogram-of-oriented-gradients detector with a linear support vector machine, together with dlib’s facial landmark detector, to crop each face to 96 by 96 pixels and each mouth region to 28 by 28 pixels from frames extracted every 35 milliseconds. A small five-layer convolutional network then classifies each mouth as open or closed, thresholding the predicted probability of an open mouth at 0.5. This mouth-state gate is the distinctive feature of the design: when the mouth is closed across a buffered sequence of ten frames, only the visual pathway runs; when the mouth is open in any frame, audio enters the pipeline.

In the audio-visual branch, speech segments are transformed in real time with a short-time Fourier transform into spectrogram images of 400 by 400 pixels, using a 256-sample window, a hop size of six samples at a 48 kHz sampling rate, and a Tukey window, yielding 129 frequency bins rendered on a decibel scale. Parallel convolutional neural network and long short-term memory, or LSTM, branches process the face-image and spectrogram sequences, each with nine convolutional layers, three pooling layers, batch normalization, and an LSTM layer with 20 percent dropout. The two representations are concatenated and passed to a softmax layer over the three sentiment classes. Predictions from the tenth frame of each face track onward are averaged to produce a single sentiment label for the track. The entire architecture contains 20,897,598 trainable parameters.

The researchers trained and evaluated the system on the Ryerson Audio-Visual Database of Emotional Speech, or RAVDESS, which contains 24 professional actors speaking with a neutral North American accent. Because categories such as surprise, disgust, and fear have valence that shifts with context, the team grouped the eight emotion labels into three sentiment classes: calm and neutral became neutral, happy became positive, and angry and sad became negative. Each entire video received a single clip-level label, a choice the authors acknowledge introduces label noise when a clip contains segments of more than one underlying sentiment. The data were split four to one into training and validation sets, with a held-out test set of 192 videos from six actors not seen during training, distributed as 48 positive, 48 neutral, and 96 negative clips.

The experiments proceeded in stages. Audio-only and visual-only convolutional networks managed only about 43 to 46 percent accuracy, showing that neither modality alone carries enough signal. Fusing the two modalities lifted the audio-visual CNN to 69.79 percent on the test set. Adding temporal modeling with LSTMs proved decisive: the audio-only CNN-LSTM reached 86.98 percent validation accuracy, and the fused audio-visual CNN-LSTM with mouth classification attained 100 percent training accuracy, 91.30 percent validation accuracy, and the headline 84.89 percent test accuracy, an improvement of roughly 15 percentage points over the non-temporal fused model. Ablations confirmed the design choices: removing mouth classification dropped accuracy to 82.29 percent, lengthening the temporal window from 10 frames to 50 frames reduced it to 75 percent, and an alternative mouth-gating scheme reached only 72.91 percent. Short windows appear to work better because they generate more training sequences and capture local affective dynamics.

The paper also reports a systematic comparison with nine large language models, spanning gpt-4o, gemini-2.5-pro, llama4:scout, deepseek-v2.5:236b, three vision-language models, and two mid-tier text models, evaluated on the RAVDESS validation images under zero-shot and three-shot prompting and two decoding configurations. On face images alone, the LLMs were competitive: gemini-2.5-pro reached 69.9 percent accuracy and gemma3:27b reached 70.8 percent, exceeding the paper’s CNN baseline of 45.92 percent and its visual-only CNN-LSTM baseline of 63.04 percent. But performance collapsed on spectrogram-only input, where the best model, deepseek-v2.5:236b, managed just 43.9 percent, and on the combined face-plus-spectrogram task, where llava:34b led at 44.8 percent and the remaining models ranged from 13.7 to 33 percent. No LLM approached the specialized framework’s 91.30 percent validation accuracy on the joint task, suggesting that current general-purpose models struggle to interpret low-level acoustic representations and to reason over synchronized audio-visual streams.

As a preliminary real-world check, the team deployed the system on a live feed from a Hikvision network camera connected to a laptop-based inference pipeline, evaluating 20 webcam videos from two actors. Accuracy fell to 60 percent overall, with a striking failure on positive sentiment: none of the five positive clips was classified correctly, while nine of ten neutral and three of five negative clips were. The authors attribute this gap to the controlled conditions of RAVDESS, where Western professional actors produce sustained, exaggerated expressions under uniform lighting and minimal background noise, in contrast to the natural facial motion, varied backgrounds, and acoustic complexity of the webcam recordings. They explicitly caution that the small size of this out-of-sample set does not support conclusions about generalization.

The authors are candid about other limitations. The reported results come from a single train-validation split rather than the speaker-independent k-fold cross-validation common in the RAVDESS literature; because clips from the same actors appear in both training and validation sets, the 91.30 percent validation figure is likely optimistic, and 84.89 percent should be read as a point estimate from the speaker-independent six-actor test set. The final model also shows signs of overfitting, with training accuracy saturating near 100 percent while validation loss stops decreasing after about 30 epochs, and regularization limited to dropout on the LSTM layers. Runtime profiling on an Apple M4 Max laptop measured a median inference latency of 34.4 milliseconds for one ten-frame sequence on the GPU, comparable to the 35 millisecond frame interval, but this covers model inference only; end-to-end throughput of the full pipeline, including face detection and spectrogram computation, was not established.

Against prior work, the comparison is careful rather than triumphal. Most RAVDESS studies address the finer eight-class emotion recognition task, which is not directly comparable to three-class sentiment. Within the sentiment task, the team’s model outperforms reproduced baselines including a VGG16-GRU style architecture with more than 270 million parameters, while the proposed network’s 20.9 million parameters make it an order of magnitude smaller. The researchers have released their prepared face and spectrogram dataset to support independent verification, and they identify speaker-independent cross-validation, improved fusion strategies, soft mouth-state gating, and evaluation on larger and more diverse out-of-sample sets as priorities for future work. For now, the study stands as evidence that compact, task-specific multimodal architectures still hold a clear edge over general-purpose LLMs for real-time audio-visual sentiment analysis.

Subject of Research: A lightweight multimodal CNN-LSTM framework for real-time audio-visual sentiment classification from video streams, benchmarked against large language models on RAVDESS.

Article Title: Lightweight audio-visual sentiment analysis with dynamic mouth-state modeling and multimodal LLM benchmarking

Article References: Yakaew, A., Racharak, T., Dailey, M. N., & Jearanaiwongkul, W. (2026). Lightweight audio-visual sentiment analysis with dynamic mouth-state modeling and multimodal LLM benchmarking. Frontiers in Artificial Intelligence, 9, Article 1855069. https://doi.org/10.3389/frai.2026.1855069

Image Credits: AI Generated

DOI: 10.3389/frai.2026.1855069

Keywords: sentiment analysis, audio-visual fusion, CNN-LSTM, RAVDESS, multimodal large language models, mouth-state classification, real-time video, spectrograms, temporal modeling, affective computing, lightweight neural networks, Frontiers in Artificial Intelligence