Hybrid Transformer Model Reads Text and Images Together to Flag Hate Speech

Hate speech on social media rarely announces itself with a slur. It hides in memes, sarcastic captions, and image-text pairings where the hostility emerges only when words and visuals are read together. A study published on 28 October 2026 in Discover Computing presents a context-aware hybrid multimodal transformer framework that combines contrastive annotation, cross-modal transformer fusion, disagreement-aware label refinement, graph-based lexical expansion, and ordinal hate-intensity scoring to address exactly this problem. The work, led by Rakesh Bharati, Jyoti Bharti, and Vasudev Dehalwar with Rijvan Beg, reports consistently higher accuracy, precision, recall, and F1-scores than the reference methods it was compared against.

The framework rests on five integrated modules. The Dual-Modality Polarity-Guided Contrastive Annotation module, or DMPCA, builds a joint polarity-emotion vector that blends the sentiment tendency of a text with the emotional content of its associated image. A contrastive loss based on cosine similarity pulls matched text-image pairs closer in a shared embedding space while pushing mismatched pairs apart. Importantly, the authors note that matched pairs are not assumed to share identical polarity and emotion in every case; when the two modalities diverge substantially, the sample is retained as a potentially informative incongruent case rather than forced into alignment. Such incongruity is precisely what characterizes sarcastic, ironic, and meme-based hate speech.

The second module, the Contextual Lexical-Visual Transformer (CLVT), fuses the two streams through bidirectional cross-attention. Textual embeddings come from a BERT-base encoder with twelve attention heads and a hidden dimension of 768, while visual embeddings are produced by a ResNet-50 backbone pretrained on ImageNet and fine-tuned on AffectNet. The authors are explicit that AffectNet is designed primarily for facial affect recognition and therefore does not fully capture symbolic, textual, or culturally dependent emotions in memes, so its features are treated as auxiliary affective cues rather than definitive meme-emotion labels. Four encoder-decoder layers with eight attention heads each align textual polarity with visual emotion in the semantic embedding space.

Annotation quality is tackled by the Pseudo-Human Feedback Loop with Disagreement Resolution Network (PHL-DRN). An ensemble of three models computes the entropy of its output, and samples exceeding an empirically set entropy threshold of 0.75 are routed to a re-annotation step in which the consensus prediction is taken. This disagreement-aware refinement targets the well-known problem that hate speech labels are subjective, culturally contingent, and often inconsistent across annotators, which degrades models that assume annotations to be accurate.

The Lexical Hate Amplifier Graph (LHAG) addresses keyword blindness. It constructs a similarity-based word graph whose edges are weighted by cosine similarity, then propagates initial hate-lexicon labels through the graph using a normalized adjacency propagation scheme for twenty iterations with a similarity threshold of 0.7. This spreads hate-related signals to semantically related but lexically distant words, allowing the model to detect coded or emerging hate expressions that static keyword lists and lexicon-based approaches miss.

Finally, the Multi-Granular Evaluation with Hate Density Scoring module replaces binary hate/non-hate decisions with an ordinal scale. A per-sentence hate density score, averaged over word-level scores bounded between zero and one, is converted into four severity categories—Non-Hate, Mild Hate, Moderate Hate, and Extreme Hate—using thresholds. The authors acknowledge a limitation here: aggregating local scores may not fully preserve compositional phenomena such as negation, sarcasm, irony, and long-range sentence dependencies, and they plan to condition severity estimation directly on sentence-level transformer representations in future work.

The experimental corpus comprised 20,000 samples: 12,000 text-only samples drawn from HateXplain and the Jigsaw Toxic Comments dataset, and 8,000 text-image pairs combining Facebook Hateful Memes material with 3,000 multimodal posts collected from Twitter and Reddit and annotated in-house. The authors stress that the 20,000 figure represents a curated corpus, not newly collected data, and that the original annotation contribution is the 3,000 in-house samples. Those samples were labeled independently by annotators, inter-annotator agreement was calculated, and disputed cases were resolved by consensus adjudication before model development. In a 1,000-sample subset, hate density scores agreed with human judgment at 90.3 percent accuracy.

Trained on an NVIDIA A100 GPU with PyTorch 2.1.0, using an AdamW optimizer with a learning rate of 1e-5 for BERT layers and 5e-5 for newly initialized layers, and split 70/15/15 into training, validation, and test sets, the full model achieved 91.8 percent accuracy, a macro-F1 of 89.7 percent, precision of 91.1 percent, recall of 88.2 percent, and an AUC-ROC of 94.5 percent on the combined dataset. Across five independent runs with different random seeds, mean accuracy was 91.74 percent with a standard deviation of 0.41, and paired t-tests comparing the proposed model with each baseline yielded p-values consistently below 0.01 for accuracy, macro-F1, and AUC, with Levene’s test supporting variance equivalence.

The comparison deserves careful framing, and the authors provide it themselves. The three baselines—Method 3, an optimized deep-learning approach for multilingual hate speech detection; Method 9, a feature-level bias and fairness classifier; and Method 26, an Ebola-optimized spiking neural network—were chosen as methodologically diverse reference points rather than structurally equivalent multimodal architectures. None jointly incorporates multimodal fusion, disagreement-aware annotation refinement, graph-based lexical expansion, and ordinal severity estimation. The reported gains of 2.6 to 7.5 percent are therefore presented as reference comparisons, not as a direct state-of-the-art multimodal benchmark, a limitation the paper explicitly acknowledges.

Ablation studies support the contribution of each module: removing any single component reduced macro-F1 by 2.8 to 6.4 percent, with contrastive annotation and cross-modal transformer removal causing the largest drops. Cross-modal generalization tests showed the model remained stable under modality drop-out, where several rival multimodal approaches degrade sharply. On the efficiency side, inference took 85 to 95 milliseconds per multimodal post on a GPU, under 40 milliseconds for text-only input, and batch inference exceeded twelve posts per second. Human evaluators, with inter-rater agreement above 0.82, rated explanation coherence at an average of 4.5 on a five-point Likert scale, and annotation consistency on borderline samples reached 92.1 percent. The authors caution that attention-map explanations may not always coincide with causal linguistic structures, which matters in legal or policy contexts, and that pre-trained encoders may struggle with slang and emerging hate symbols unless further fine-tuned. Future work targets low-resource and cross-lingual settings, including Vietnamese using the ViHSD and ViHOS datasets, multilingual deployment, and extension to audio and OCR-extracted text from video platforms.

Subject of Research: A hybrid multimodal transformer framework for detecting hate speech in text and images using contrastive annotation, graph-based lexical expansion, and ordinal severity scoring.

Article Title: A context-aware hybrid multimodal transformer framework for hate speech detection via contrastive annotation and graph-based semantic Expansion

Article References: Bharati, R., Bharti, J., Dehalwar, V., & Beg, R. (2026). A context-aware hybrid multimodal transformer framework for hate speech detection via contrastive annotation and graph-based semantic Expansion. Discover Computing, 29(1), Article 823. https://doi.org/10.1007/s10791-026-10776-9

Image Credits: AI Generated

DOI: 10.1007/s10791-026-10776-9

Keywords: hate speech detection, multimodal transformer, cross-modal attention, contrastive learning, social media moderation, graph label propagation, BERT, ResNet-50, annotation disagreement, hate severity scoring, HateXplain, natural language processing