SPC-Verify Framework Detects Adversarial Attacks on Image Captioning Models

Researchers from Yunnan University School of Government in Kunming, China, have introduced SPC-Verify, a Semantic–Phenomenon Consistency Verification framework for adversarial detection in image captioning. As detailed in an original research article published in the journal Frontiers in Computer Science under the Computer Security section, the framework confronts an acute security vulnerability in cross-modal artificial intelligence models that generate natural language from visual imagery. Image captioning models remain susceptible to carefully crafted adversarial perturbations that can manipulate their computational outputs, causing them to generate descriptions that conflict directly with visual evidence, violate established physical rules, or contradict foundational commonsense knowledge about the world.

Conventional detection methodologies developed for machine learning models have predominantly depended on identifying confidence anomalies, tracking feature-space deviations, or measuring reconstruction errors. These existing detection methods are not well suited to cross-modal generative tasks. A primary technical difficulty in defending these generative systems lies in jointly evaluating localized token-level visual-semantic consistency alongside global sentence-level plausibility.

To resolve these structural limitations, the authors structured the SPC-Verify architecture around two complementary verification branches: Visual-Semantic Consistency Verification and Phenomenon Plausibility Verification. The first branch, Visual-Semantic Consistency Verification, examines whether token-level visual evidence guided by localized heatmaps actually supports the specific linguistic concepts generated by the captioning model.

To enhance the fidelity and reliability of local visual evidence during this token-level inspection, the researchers created SCAR-SAT, a post-hoc heatmap generation and refinement method specifically tailored for Show-Attend-and-Tell-based captioning architectures. SCAR-SAT refines token-level visual grounding by systematically integrating four complementary properties: semantic consistency, lightweight counterfactual support, spatial continuity, and temporal stability.

The second core component of the framework, Phenomenon Plausibility Verification, operates at the complete sentence level rather than inspecting individual tokens in isolation. While the visual-semantic branch verifies that generated words correspond directly to local image regions, Phenomenon Plausibility Verification assesses whether the full caption adheres to real-world physical constraints and commonsense knowledge.

In addition to the dual verification branches, the researchers addressed model transferability by introducing Zero-Shot Cross-Model Threshold Transfer, abbreviated as ZS-CMTT. The proposed threshold transfer method learns shared decision thresholds directly from the joint score distributions of two established captioning architectures, Show-Attend-and-Tell and BLIP. These learned decision thresholds can then be transferred directly to unseen captioning models without target-model samples.

Together, these components provide an interpretable and transferable framework for adversarial detection in image captioning.

Subject of Research: Adversarial example detection in image captioning models

Article Title: Adversarial example detection for image captioning models via SPC-verify

Article References: Zhang, D., Mao, J., & Zhu, J. (2026). Adversarial example detection for image captioning models via SPC-verify. Frontiers in Computer Science, 8, Article 1897185. https://doi.org/10.3389/fcomp.2026.1897185

Image Credits: AI Generated

DOI: 10.3389/fcomp.2026.1897185

Keywords: Adversarial Example Detection, Image Captioning, Computer Security, Cross-Modal Verification, Semantic Consistency, Phenomenon Plausibility, SCAR-SAT, Show-Attend-and-Tell, BLIP, Threshold Transfer, Adversarial, example