Robots Reading Feelings: Review Reveals Lab Gains Fade Fast in Real-World Emotion Recognition

Robots that can tell whether a person is happy, stressed, or frustrated have long been a staple of science fiction, but a new systematic review shows how far the field has come and how far it still has to go. A literature review published in Discover Robotics synthesizes 42 empirical studies from 2015 to 2025 on speech, facial, and multimodal emotion recognition tailored for robotic contexts. The authors, led by Nikhilsingh Parihar with colleagues including Giancarlo Fortino, classified methods by input modality, catalogued the deep learning architectures and signal processing strategies behind them, and measured how well each approach survives the messy conditions of actual deployment.

The headline finding is a sobering gap between laboratory and real-world performance. Across the reviewed literature, reported accuracies ranged from 68 to 96 percent under controlled conditions, but real-world deployments typically lost 4 to 27.5 percent of that performance. When the authors computed paired lab and real-world scores from the studies reporting both, speech emotion recognition systems degraded by a mean of 8.6 percent, with a standard deviation of 3.0 percent, while facial emotion recognition systems dropped by a mean of 14.4 percent, with a standard deviation of 7.1 percent. The higher variance in facial systems reflects their greater sensitivity to deployment-specific factors such as lighting and camera pose, relative to the acoustic variability that plagues speech systems.

The review also documents a decisive shift toward multimodal approaches. Multimodal systems accounted for 53 percent of the studies, or 22 of the 42, while facial-only systems made up 26 percent with 11 studies and speech-only systems 21 percent with 9 studies. The authors attribute this trend to the recognition that combining multiple input modalities improves robustness and accuracy in challenging real-world environments, though they caution that sensor integration, computational cost, and environmental variability remain significant barriers to deployment.

Speech emotion recognition draws on prosodic features such as pitch, energy, and speaking rate, along with spectral patterns, and is prized for being non-intrusive and functional even when the user is not in direct visual contact with the robot. Traditional pipelines relied on handcrafted features with shallow classifiers such as support vector machines, Gaussian mixture models, and hidden Markov models, but recent work has moved to end-to-end deep learning using convolutional neural networks, recurrent networks, BiLSTMs, and transformer architectures. Among the reviewed studies, lab accuracies for speech systems ranged from 72 to 96 percent while real-world deployments achieved between 68 and 85 percent, with degradation driven by speaker-robot distance, reverberation, robot-generated ego-noise, and overlapping dialogue.

Several studies illustrate the engineering required to make speech work on real robots. Grageda and colleagues employed ladder networks with beamforming for distant speech emotion recognition on the PR2 robot, achieving up to a 180 percent improvement in Concordance Correlation Coefficient under simulated acoustic conditions, and a follow-up study integrated wav2vec 2.0 embeddings to improve robustness under dynamic robot movement. Garcia and colleagues demonstrated that self-attentive MVDR combined with multitask learning significantly outperformed traditional beamforming under noisy deployment scenarios. Other notable architectures include BiRNN-GRU models with 1D CNN layers and attention-based BiLSTM models leveraging deep spectrum representations, while personalized approaches have used Vision Transformer-based models trained on speaker-specific dialogue with NAO robots to adapt to individual vocal characteristics in low-resource settings.

Facial emotion recognition tells a different story. These systems often reach classification accuracies above 90 percent in lab-controlled environments when trained on datasets such as FER2013, CK+, or AffectNet, and modern approaches use CNNs, recurrent networks, and their hybrids to learn spatial and temporal features robust to varied expressions, head poses, and illumination. But real-world human-robot interaction introduces occlusion, movement-induced blur, sensor misalignment, and inconsistent lighting that lab setups do not face. One striking example comes from Ruiz-Garcia and colleagues, who developed a CNN-LSTM hybrid deployed on the NAO robot that achieved 96.26 percent accuracy under lab conditions but dropped to 68.75 percent during real-world deployment, a decline attributed to pose shifts and visual noise. Other work has benchmarked models such as DeepFace and CNN classifiers in industrial settings, emphasizing trade-offs between model complexity and real-time inference on embedded platforms, and has embedded continuous facial feedback into closed-loop control systems that adjust robotic behavior in collaborative tasks by predicting valence and arousal in real time.

Multimodal fusion, the review’s fastest-growing category, can occur at the feature level, the decision level, or at intermediate representations using attention-based mechanisms. Early fusion enables joint learning from raw or low-level features across modalities, allowing richer inter-modal correlations, but it is sensitive to temporal misalignment and requires carefully synchronized data streams. Late fusion combines decisions from unimodal classifiers, which is advantageous when modalities operate at different frequencies or when partial data is missing. Representative studies include real-time multimodal systems on the Pepper robot integrating speech and visual cues in assistive scenarios, and late fusion of facial expressions with contextual cues to support elderly care through adaptive dialogue strategies. Beyond behavioral signals, physiological measures such as EEG, ECG, and galvanic skin response offer direct insight into internal affective states, though sensor calibration and motion artifacts significantly limit their robustness during mobile or real-time deployment, and contact-based wearable sensors constrain naturalistic use in open-world settings.

The authors identify a set of persistent technical barriers that cut across modalities. Temporal synchronization of misaligned data streams, such as EEG and speech, can degrade performance especially in early fusion setups. Real-time fusion of high-dimensional signals such as video combined with EEG requires processing power often beyond lightweight robotic platforms. Most public datasets are unimodal and not tailored for human-robot collaboration, making collection of synchronized, annotated multimodal corpora labor-intensive and application-specific. To counter these problems, adaptive and hierarchical fusion methods are gaining traction, dynamically weighting modalities based on signal quality, user behavior, or environmental context. One representative uncertainty-aware framework computes modality weights at runtime from quality signals such as signal-to-noise ratio for speech or face-detection confidence for facial recognition, using a softmax function with a temperature parameter, and couples these weights with a negative-entropy regularizer in the training loss to prevent degenerate collapse onto a single modality.

Looking forward, the review outlines several research priorities. Fewer than 20 percent of studies explicitly evaluated models across diverse environmental or cross-device settings, pointing to a need for domain-invariant learning through adversarial domain adaptation and contrastive learning, unsupervised and test-time adaptation, and cross-robot generalization so that models trained on one platform such as NAO transfer to others like Pepper or PR2. Only a small fraction, between 10 and 15 percent, of existing systems incorporate user-specific or culturally aware modeling, despite the subjectivity of emotion expression, suggesting that few-shot and meta-learning, continual learning, and cross-cultural datasets deserve greater attention. On the systems side, model compression through pruning, quantization, and knowledge distillation can reduce inference time and memory footprint, while on-device learning through federated approaches or local fine-tuning enables continual updates without central data sharing. Diffusion-based synthesis, as demonstrated by CP-Diffusion, offers a route to augmenting scarce training data with realistic, prompt-conditioned emotional video sequences.

Ethical and societal considerations round out the review’s agenda. Fewer than 10 percent of the reviewed studies explicitly addressed privacy or fairness, even as emotion-aware systems raise concerns around surveillance, emotional manipulation, and algorithmic fairness. Facial analysis systems have documented accuracy disparities across gender and skin tone, speech systems trained predominantly on English-language corpora risk systematic underperformance for non-native speakers, and physiological signals such as EEG and heart rate variability are biometric by nature, requiring explicit informed consent and data minimization protocols. The authors argue that future frameworks must prioritize privacy-preserving learning through federated or differentially private approaches, fairness auditing across gender, age, and cultural subgroups, and transparency so that emotion-based decisions in robots remain interpretable, particularly in sensitive domains such as elder care and education. The consistent theme, they conclude, is that emotion-aware robots have the potential to transform healthcare, education, and assistive living, but their societal acceptance will depend on building systems that are not only intelligent but also inclusive, transparent, and respectful of human values.

Subject of Research: Emotion recognition methods for human-robot collaboration

Article Title: Emotion recognition in human robot collaboration for multimodal approaches, real-world challenges and future directions

Article References: Parihar, N., Kanan, Chawla, R., & Fortino, G. (2026). Emotion recognition in human robot collaboration for multimodal approaches, real-world challenges and future directions. Discover Robotics, 2(1), Article 13. https://doi.org/10.1007/s44430-026-00028-2

Image Credits: AI Generated

DOI: 10.1007/s44430-026-00028-2

Keywords: emotion recognition, human-robot collaboration, multimodal fusion, speech emotion recognition, facial emotion recognition, deep learning, domain adaptation, personalization, edge computing, review study, Emotion, recognition