A new multimodal artificial intelligence system has demonstrated that combining speech, language, and visual information can substantially improve the recognition of human emotions. Developed by researchers Erwin Budi Setiawan and Arliyanna Nilla at Telkom University in Indonesia, the system achieved an accuracy of 86.48% when classifying five emotional states: angry, excited, frustrated, neutral, and sad. The results, reported in the Journal of Big Data, suggest that emotion-aware machines may become more reliable when they are designed to interpret several kinds of human signals at the same time rather than depending on a single source of information.
Emotion detection is a central challenge in human–computer interaction because feelings are rarely expressed through one channel alone. A person’s words may sound neutral while their facial expression communicates frustration, or an apparently cheerful statement may be delivered in an angry tone. Systems that analyze only text, audio, or images can therefore miss important context. The researchers addressed this limitation by constructing a unified deep learning framework that processes multiple forms of data before making a final prediction. Their approach was tested using the IEMOCAP dataset, a widely used benchmark containing acted conversations in which participants express a range of emotional states through speech, language, and visual behavior.
The framework begins by converting audio into linguistic information through transcription. Rather than relying primarily on acoustic properties such as pitch, volume, or speaking speed, the system uses the words spoken in the audio recordings as a major source of emotional evidence. This design allows the language-processing component to examine the semantic meaning of an utterance. The transcription is then enriched with three complementary language technologies: RoBERTa, FastText, and the NRC Emotion Lexicon. Together, these tools provide the model with contextual, lexical, and emotion-related information, helping it distinguish between words whose emotional meaning can change depending on how they are used.
RoBERTa is a transformer-based language model designed to interpret words in relation to their surrounding context. This is important for emotion recognition because the same term can communicate different feelings in different sentences. FastText adds another layer by representing words and their subword components, which can help the system handle vocabulary variations and less common word forms. The NRC Emotion Lexicon contributes explicit links between words and emotional categories, giving the model a structured resource that connects language with affective meaning. The researchers’ hybrid enrichment scheme combines these approaches instead of treating them as competing alternatives, creating a richer textual representation before the information is sent to the fusion stage.
The visual component uses ResNet-18, a convolutional neural network architecture developed for extracting features from images. In this system, ResNet-18 converts visual input into numerical representations that capture patterns potentially associated with emotion, including facial configurations and other image-level cues. The network does not simply store a picture; it transforms visual information into a feature vector that can be compared and combined with representations derived from language. This is a crucial step in multimodal learning, because audio, text, and images are expressed in different mathematical forms. Feature extraction creates a common computational basis on which the separate signals can be integrated.
After the individual modalities have been processed, the system applies feature fusion to combine their representations. The fused features are passed to a multilayer perceptron, or MLP, which serves as the final classifier. An MLP is a feed-forward neural network made up of interconnected layers that learn how combinations of input features correspond to output categories. In this case, it learns patterns linking language-based evidence and visual cues to the five target emotions. The architecture is intended to compensate for weaknesses in any single modality. If the wording of a sentence is ambiguous, visual information may help; if an image is unclear, the textual content may provide stronger evidence.
According to the study, the multimodal configuration produced an accuracy of 86.48%, approximately 7.48 percentage points higher than the best unimodal model evaluated by the researchers. The system also achieved a macro F1-score of 0.8690. The F1-score combines precision, which measures how often a predicted category is correct, and recall, which measures how many examples of that category are successfully identified. Macro-averaging calculates these results across categories and then gives each class equal weight. This matters when a dataset contains imbalanced emotional categories, because a model could otherwise appear successful by performing well on common emotions while neglecting less frequently represented ones.
The researchers also conducted a statistical comparison between the multimodal architecture and a baseline model. The reported Z-value was 16.94, with significance stated at p < 0.05, supporting the conclusion that the observed improvement was not merely the result of random variation under the study’s evaluation procedure. The finding strengthens the central claim that integrating three modalities can make a meaningful contribution to emotion classification. However, the system’s performance should still be interpreted within the boundaries of the IEMOCAP dataset and its specific experimental design. Emotion is influenced by culture, social setting, individual personality, and spontaneous behavior, all of which can be difficult to capture in controlled or curated data.
The study arrives as researchers and technology companies search for more natural forms of interaction between people and machines. Emotion-sensitive systems could eventually support conversational assistants, educational software, accessibility tools, mental-health interfaces, customer-service platforms, and social robots. A system capable of detecting frustration might adjust its explanations, while one recognizing confusion or sadness could alter the tone and pace of its responses. Yet these possibilities also raise important concerns about privacy, consent, bias, and misinterpretation. Emotional states cannot be measured directly from a face, voice, or sentence with absolute certainty, and automated predictions should not be treated as definitive judgments about a person’s inner experience.
By combining transformer-based language analysis, word-embedding technology, an emotion lexicon, convolutional image processing, and neural feature fusion, the Telkom University team presents a technically integrated route toward more capable affective computing. Its results indicate that machines can gain a clearer statistical picture of emotion when they examine several signals together. The next challenge will be to determine how well such systems perform on spontaneous, culturally diverse, real-world interactions and whether they can explain the reasoning behind their predictions. For now, the findings offer a compelling demonstration that the future of emotion-aware artificial intelligence may depend less on teaching machines to read one signal perfectly than on teaching them to interpret many imperfect signals together.
Subject of Research: Multimodal deep learning for human emotion detection
Article Title: A multimodal deep learning system for enhanced emotion detection
Article References: Budi Setiawan, E., & Nilla, A. “A multimodal deep learning system for enhanced emotion detection.” Journal of Big Data (2026).
Image Credits: AI Generated
DOI: 10.1186/s40537-026-01476-8
Keywords: Emotion detection, multimodal learning, deep learning, IEMOCAP, feature fusion, ResNet-18, RoBERTa, macro-averaging, statistical significance, affective computing
Tags: deep learning for emotion classificationemotion recognition accuracy improvementemotion-aware artificial intelligence systemsfacial expression and speech emotion analysishuman emotion detection using AIIEMOCAP dataset emotion benchmarkintegrating speech and visual cues for emotion detectionmulti-source emotional signal processingmultimodal emotion recognitionmultimodal human-computer interactionreliable multimodal emotion classificationspeech and visual emotion analysis
