PSO fusion reaches 94% accuracy on television sarcasm benchmark

A multimodal sarcasm detector that combined linguistic, vocal and visual signals with Particle Swarm Optimization (PSO) reached 94.0% accuracy on the MUStARD benchmark, according to a computational model-development study using 690 annotated television-dialogue utterances. The model used RoBERTa to interpret transcripts, HuBERT to represent speech and ViViT to capture facial expressions and gestures. Rather than assigning the three feature streams fixed weights, the researchers searched for a global weighting configuration that minimized validation loss, then passed the weighted representation through a two-layer fully connected classifier.

The system produced a weighted F1 score of 0.940, compared with 0.920 for a directly evaluated learnable-weight baseline, although that difference did not reach statistical significance in the reported paired test (p = 0.0625). The experiment used MUStARD, a public dataset of synchronized video, speech and transcripts extracted from television sitcoms including Friends, The Big Bang Theory and The Golden Girls. Each sample is associated with an utterance labeled as sarcastic or non-sarcastic, with preceding dialogue retained as context.

The 690 samples are evenly divided between the labels, comprising 345 clips in each class. Utterances were split at the sample level into 414 training examples, 138 validation examples and 138 test examples, with stratified sampling intended to preserve class balance. The researchers reported three runs with different random seeds and presented the mean and standard deviation for the principal metrics.

This was a benchmark study, not a clinical trial or a field deployment of a sarcasm-detection system. Sarcasm was treated as a cross-channel inference problem because a transcript may omit irony carried by a voice, while a facial expression or change in intonation can alter the apparent meaning of ordinary words. The text branch used RoBERTa-Base to produce contextual embeddings for the target utterance and its conversational history.

Preprocessing retained negation, intensifiers and emphasis markers because they can change pragmatic meaning, while informal forms such as “gonna” and “wanna” were normalized and inputs were tokenized with byte-pair encoding. The final hidden representation came from the model’s classification token and entered a binary sarcasm classifier. On its own, this branch achieved 0.82 accuracy, with recall and precision both reported as 0.82 and a weighted F1 score of 0.82.

Sound supplied a different view of the same utterance. The audio pipeline converted speech to waveform files, resampled them to 16 kHz, normalized amplitude, filtered background noise, removed non-speech intervals with a voice-activity detector and standardized utterance duration through truncation or zero-padding. A pretrained HuBERT-Base checkpoint converted raw waveforms into frame-level features with a convolutional encoder, then used stacked self-attention layers to model contextual acoustic patterns.

Lower layers were frozen while upper layers were fine-tuned, and mean pooling produced a 768-dimensional utterance representation. The resulting audio-only model reached 0.88 accuracy, recall and precision, with a weighted F1 score of 0.88. Its class-level errors were nearly balanced: the confusion matrix contained 63 correct non-sarcastic decisions and 59 correct sarcastic decisions, alongside eight errors in each direction.

The visual branch used ViViT-Base to model the timing as well as the appearance of facial and gestural cues. Video was sampled at 25 frames per second, with faces detected and cropped to 224-by-224-pixel regions. Pixel values were normalized, histogram equalization reduced some illumination differences, and augmentation included horizontal flipping, rotation and brightness adjustment.

Sixteen-frame clips were supplied to the transformer, whose patch embeddings and self-attention captured spatial relationships within frames and temporal dependencies across frames. This model alone obtained 0.90 accuracy and a weighted F1 score of 0.89, making it the strongest unimodal result in the study. It was also conservative about labeling utterances as sarcastic: 61 non-sarcastic and 63 sarcastic examples were correctly classified, with seven false positives and seven false negatives.

The central technical contribution was a compact, derivative-free search over the relative contribution of the three representations. Each particle in the PSO population represented a candidate vector containing text, audio and video weights. The weights were constrained to be non-negative and to sum to one.

Particles moved through the search space using personal and global best positions, and their positions were renormalized at each update. Candidate vectors were evaluated on the validation set, after which the best configuration was applied globally to the fused feature representation. The implementation used 30 particles and allowed up to 50 iterations, with convergence reported around 30 to 40 iterations.

The encoder parameters were held fixed during this stage, so the optimization changed three modality weights rather than the millions of parameters inside the three pretrained transformers. The method is feature-level rather than decision-level fusion. Modality embeddings are projected into a shared feature space and combined as a weighted sum before the final classifier makes a sarcastic or non-sarcastic prediction; individual class probabilities are not averaged or voted together.

It also differs from sample-adaptive reliability weighting because the optimized weights are learned globally from the validation set and remain fixed for all test samples and subsequent inference. The authors describe the approach as model-agnostic because the fusion module can be attached to different pretrained encoders without changing their internal architecture, but its reported benefits were demonstrated only with RoBERTa, HuBERT and ViViT on MUStARD. Controlled fusion experiments showed that the optimization procedure contributed to the gain.

With the same encoders and experimental protocol, simple concatenation produced a weighted F1 score of 0.88, equal-weight fusion produced 0.90 and learnable weighted fusion produced 0.92, while PSO-based fusion reached 0.94. The reported difference from the directly evaluated learnable baseline was two percentage points, but the paired statistical comparison did not pass the study’s 0.05 significance threshold. Ablation results also showed that the three channels were complementary.

Audio plus video was the strongest two-channel combination, with accuracy around 91.1% to 91.5% across the reported summaries, while audio plus text reached about 83.5% and text plus video about 84.5%. The full audio, text and video model reached approximately 94.4% in the modality comparison and 94.0% in the main evaluation. The results must be interpreted within the benchmark’s limitations.

Although the paper initially describes a speaker-independent protocol, the data were ultimately partitioned at the utterance level, so speakers and dialogue scenes could recur across the training, validation and test sets. The authors explicitly acknowledge that complete speaker- and dialogue-independent generalization was not guaranteed. The study did not quantify parameter count, floating-point operations, peak memory, training time or inference latency, making claims of real-time efficiency or lightweight deployment unverified.

Robustness tests nevertheless showed a gradual performance decline as modalities were omitted or corrupted, with accuracy ranging from 0.878 under 30% text corruption to 0.940 with intact inputs, underscoring the continued value of complementary evidence. Among the eight final test errors, the researchers classified three as context-dependent sarcasm, two as subtle or weak visual cues, two as cross-modal inconsistency and one as weak acoustic or prosodic cues. These categories identify the remaining challenge: some sarcastic intentions are not expressed through any conspicuous channel and must be inferred from conversational history, shared knowledge or culturally specific references.

The study supports a clear but bounded conclusion: on one balanced benchmark of television dialogues, explicitly optimizing the contribution of text, speech and video produced the strongest reported performance, with 94.0% accuracy and a 0.940 weighted F1 score. The evidence does not establish that the same gain will occur in larger, more diverse or fully speaker-independent datasets, nor does it show that the advantage over learnable weighting is statistically conclusive. Within the tested setting, however, the results show that sarcasm recognition benefits from combining complementary modalities and that the method of weighting them is itself an important experimental variable.

Subject of Research: Multimodal sarcasm detection using transformer encoders and Particle Swarm Optimization-based feature fusion

Article Title: Enhanced multimodal transformer framework for sarcasm detection with particle swarm optimization based feature fusion

Article References: Payala, V. K., Jeet, A. A., & Abinaya, S. (2026). Enhanced multimodal transformer framework for sarcasm detection with particle swarm optimization based feature fusion. Frontiers in Artificial Intelligence, 9, Article 1947291. https://doi.org/10.3389/frai.2026.1947291

Image Credits: AI Generated

DOI: 10.3389/frai.2026.1947291

Keywords: sarcasm detection, multimodal learning, Particle Swarm Optimization, RoBERTa, HuBERT, ViViT, feature fusion, MUStARD dataset, transformer models, conversational AI, Enhanced, multimodal