Systematic review finds attention explanations are rarely tested for faithfulness

A systematic literature review of 168 primary studies found that 57.14% left the relationship between predictive performance and attention-map faithfulness unexamined, while 41.07% reported synergy between the two objectives. Only 1.79% explicitly documented a performance–faithfulness trade-off, suggesting that negative or conflicting findings have been underrepresented in the literature.

The review, published on 25 September 2026, examined attention-based explanation faithfulness in convolutional neural networks and Vision Transformers used for image-based tasks. It focused on whether attention maps faithfully represent a model’s underlying computational logic, rather than treating human-readable plausibility and technical faithfulness as interchangeable. The included studies spanned multiple application domains, with medical imaging accounting for 35.1% of the corpus and general computer vision accounting for 22%.

Attention has two roles in the reviewed literature: it can prioritize features to support prediction and can also provide spatial or patch-level maps used for explanation. In CNNs, attention commonly appears as an auxiliary module that reweights feature maps, while in Vision Transformers it forms much of the architectural backbone for feature aggregation and global context modeling. The review separated interpretability, defined as a human-readable explanation, from faithfulness, defined by whether an explanation accurately reflects the model’s underlying causal influences.

The authors followed systematic-review guidance described by Kitchenham and Charters and used a PRISMA-inspired workflow. Searches were conducted in October 2025 across Web of Science, Scopus and IEEE Xplore for English-language, peer-reviewed work published from January 2017 through October 2025. The search produced 464 records. After removal of 74 duplicates, 390 unique records underwent title-and-abstract screening, 194 were assessed in full text, and 168 primary studies were retained. All retained studies met the review’s minimum quality threshold under five assessed criteria: architectural clarity, dual-role evaluation, faithfulness metrics, benchmarking practices and reproducibility.

Each study was coded along two design dimensions: whether attention was integrated as a modular auxiliary block or as a core computational backbone, and whether explanation was an implicit byproduct or an explicit objective. These dimensions produced four paradigms: performance-driven attention, plausibility-driven attention, human-aligned attention, and causal and counterfactual attention. Design classification was treated separately from the reported empirical result and evaluation method, so membership in a paradigm did not itself determine whether a study found synergy, a trade-off or no examined relationship.

For the performance–faithfulness synthesis, studies were coded as demonstrating synergy when they reported simultaneous gains in predictive accuracy and explanation faithfulness, as a trade-off when improving one explicitly degraded the other, and as implicit or unexamined when only one dimension was evaluated or the relationship was not measured. Coding reliability was checked by two researchers for 34 studies, approximately 20% of the corpus. Raw inter-rater agreement was 91.2% for the design classification, with Cohen’s kappa of 0.88; corresponding kappa values were 0.88 for the empirical synthesis and 0.84 for evaluation methodologies. Disagreements were resolved by consensus.

The review also identified a methodological transition from qualitative inspection of attention maps toward human-grounded alignment and perturbation-based causal tests. Qualitative inspection can establish visual plausibility but does not by itself demonstrate causal relevance. Human-grounded methods compare model attention with expert annotations or gaze data, while causal and quantitative methods use interventions such as deletion, insertion, counterfactual tests and pointing games. The authors classified unexamined evaluation as a separate category and proposed combining multiple methods rather than treating any single metric as a complete measure of faithfulness.

The review’s evidence has several limitations. It was restricted to English-language, peer-reviewed studies on image-based tasks published through October 2025, with CNNs and ViTs as its principal scope. Most included work did not directly test the performance–faithfulness relationship, and the very small number explicitly reporting a trade-off limits strong conclusions about its prevalence or causes. The review authors did not generate or analyze datasets, and the four-part taxonomy and reporting checklist are proposed syntheses of the included literature rather than experimentally validated instruments.

Overall, the synthesis suggests that attention’s apparent explanatory value cannot be inferred from predictive accuracy or from the visual appearance of an attention map. The review’s central quantitative finding is that most studies either did not examine the performance–faithfulness relationship or reported alignment between the two, while explicit evidence of conflict was rare. The authors therefore call for more rigorous, standardized evaluation of attention explanations and for designs that make faithfulness an explicit objective rather than an incidental output of accuracy-oriented optimization.

Subject of Research: Systematic review of attention-based explanation faithfulness and predictive performance in deep visual models

Article Title: A systematic review of attention duality in deep visual models for performance and faithfulness tradeoffs

Article References: Krisnandi, D., Ramli, K., Purnamasari, P. D., Pardede, H. F., & Salman, M. (2026). A systematic review of attention duality in deep visual models for performance and faithfulness tradeoffs. Discover Artificial Intelligence, 6(1), Article 1262. https://doi.org/10.1007/s44163-026-02209-9

Image Credits: AI Generated

DOI: 10.1007/s44163-026-02209-9

Keywords: attention mechanisms, explainable AI, faithfulness, vision transformers, convolutional neural networks, systematic review, performance trade-offs, causal evaluation, model interpretability, systematic, review, attention