Supervised meta-model improves human alignment of Arabic AI explanations

Human supervision can make artificial-intelligence explanations of Arabic hotel reviews more closely match the words people consider important, but the improvement comes with a measurable loss of fidelity to the model being explained. In a study of 303 Arabic TripAdvisor reviews, researchers trained three meta-models to combine several explainable artificial intelligence methods with contextual, linguistic and structural features. The best system, CatBoost, reached an F1 score of 0.84 when its selected tokens were compared with human rationales, outperforming LIME, SHAP, Integrated Gradients, DeepLIFT and an earlier unsupervised ensemble, whose scores ranged from 0.64 to 0.70.

The experiment therefore found that explanations can become more closely aligned with human judgments without becoming a more exact reconstruction of the classifier’s internal behavior. The study focused on CAMeLBERT-MSA, a 12-layer BERT-style model pre-trained on large Arabic corpora and fine-tuned for sentiment analysis. The classifier has 12 attention heads in each layer and produces 768-dimensional token embeddings.

For every token, the researchers assembled attribution scores from LIME, SHAP, Integrated Gradients and DeepLIFT, together with the earlier Ensemble XAI output. They also extracted attention distributions from the classifier’s classification token, using both attention averaged across all layers and heads and attention from the final layer. Contextual embeddings were reduced from 768 dimensions to 10, with principal component analysis and uniform manifold approximation and projection evaluated as alternatives.

The three meta-models—Logistic Regression, XGBoost and CatBoost—then learned to predict whether humans had marked each token as important. The resulting dataset contained 16,925 token-level instances drawn from 303 reviews. Human annotation supplied the central supervision.

Two annotators highlighted the words they regarded as most influential for each review’s sentiment through the XS2A web-based annotation platform. One annotated 163 reviews and the other 140, with 55 reviews assessed by both. Their average Cohen’s kappa was 0.57, indicating moderate agreement and also illustrating the subjectivity of deciding which words carry the meaning of a review.

The annotation protocol assigned a binary label to each token: important or non-important. For evaluating agreement with these rationales, the researchers extracted the highest-scoring 25 percent of tokens from each explanation and compared them with the human labels using precision, recall, F1 score and Jaccard similarity. To assess whether differences were unlikely to arise by chance, they performed 10,000 paired bootstrap resamples at the review level and calculated two-sided p-values and 95 percent confidence intervals for performance differences.

All three supervised meta-models achieved F1 scores between 0.80 and 0.84, with CatBoost performing best and obtaining a Jaccard index of 0.74. That Jaccard result represented an improvement of more than 17 percentage points over LIME, the strongest standalone explainer in that comparison. Review-level bootstrap analysis found that CatBoost’s advantages over all four individual explainers and Ensemble XAI were statistically significant on F1 and Jaccard, with every reported p-value below 0.001.

This advantage was expected because the meta-models were explicitly trained to reproduce human importance judgments, whereas the comparison methods were unsupervised post-hoc explainers that never saw those annotations. The choice of dimensionality-reduction method also mattered modestly: PCA was marginally better for Logistic Regression, while UMAP produced better results for the tree-based XGBoost and CatBoost models, indicating that nonlinear structure in the reduced representation was useful to the more flexible learners. Faithfulness was evaluated separately by asking how the explanations corresponded to CAMeLBERT’s own decisions.

Comprehensiveness measured the drop in the classifier’s confidence after tokens identified as important were removed. Sufficiency was defined as the difference between the original confidence and the confidence obtained when only the rationale was retained; lower values were treated as preferable because they indicated that a smaller set of tokens could support the prediction. Insertion and Deletion AUC tracked confidence as tokens were restored or removed in order of importance, while Kendall correlation with leave-one-out perturbation compared the explanation scores with the effect of omitting each token individually.

These five measures were integrated into a combined metric, with metrics whose preferred direction differed transformed so that higher values represented better faithfulness. The results exposed a pronounced trade-off. LIME and Ensemble XAI were more faithful to CAMeLBERT, with combined scores of 0.76, while the meta-models ranged from 0.54 to 0.61.

Logistic Regression’s comprehensiveness fell to 0.10, compared with 0.42 for Ensemble XAI and 0.41 for LIME. Ablation experiments showed that contextual embeddings carried most of the information used to approximate human judgments. When CatBoost used only reduced embedding features, it achieved an F1 score of 0.82 and a Jaccard index of 0.72, significantly outperforming the other isolated feature groups.

Linguistic and review-level features alone reached an F1 score of 0.79, XAI scores alone reached 0.77, and attention features alone produced the weakest result, with an F1 score of 0.73 and Jaccard index of 0.65. Removing embeddings from the complete system lowered F1 from 0.84 to 0.81, whereas removing XAI scores reduced it only to 0.836 and removing attention produced a smaller decrease to 0.838. Removing linguistic and review-level features lowered F1 to 0.824, showing that these signals supplied information beyond contextual embeddings alone.

Progressive fusion produced a complementary pattern: XAI-only scores began at an F1 score of 0.77, attention raised it to 0.78, embeddings increased it to 0.824, and linguistic and review-level information yielded the final F1 score of 0.842. The attribution scores thus contributed useful but limited information beyond what the transformer already encoded in its contextual representations. Qualitative examples showed how the approach performed on positive, negative and neutral reviews.

In one positive review, the meta-model concentrated on Arabic words meaning “beautiful,” “ideal” and “attracts you,” closely following the human selection and achieving an F1 score of 0.90 and a Jaccard index of 0.80. A negative example received an F1 score of 0.87 and Jaccard index of 0.76 after the model highlighted phrases meaning “I will never go,” “not clean,” “bad” and “noisy.” In a mixed neutral review, the system reached an F1 score of 0.85 and Jaccard index of 0.77 but missed the connective “but,” which marked the shift between favorable and unfavorable clauses. Other examples revealed a related weakness: the meta-model sometimes selected positive words appearing inside a negative context or emphasized sentiment-bearing terms while overlooking the classifier’s dependence on negation.

In one negative review, the supervised model and human annotators converged on words meaning “good” and “cooperative,” while LIME and Ensemble XAI focused more heavily on the negation particle that strongly affected CAMeLBERT’s prediction. The meta-model highlighted the negated term but still distributed importance across positively valenced words, consistent with a limitation in representing context and compositional relationships. Feature analysis also complicated any simple account of the results.

Logistic Regression assigned substantial importance to attention averaged across heads and layers, whereas XGBoost distributed influence across attention, embeddings and XAI scores. CatBoost ranked structural features such as token length alongside particular embedding dimensions more highly than either attention or individual attribution methods. Correlations between raw features and human annotations were generally modest: the strongest attention correlation with the target was 0.11, while token length had a correlation of −0.13.

Within the UMAP-reduced representation, several dimensions correlated strongly with sentiment polarity, suggesting that the compact embedding geometry preserved information about positive and negative language even when its direct predictive contribution appeared smaller in some models. The findings also cautioned that raw attention was not a reliable stand-alone indicator of human importance: it tended to decline in longer reviews and at later token positions, and its agreement with SHAP and Ensemble XAI scores was limited. The evidence comes with important limits.

The annotations came from 303 hospitality reviews and two annotators, with only 55 reviews jointly labeled, so the 16,925 token instances are not statistically independent and may not represent rarer dialects, other tasks or more varied domains. Each token was treated separately, preventing the framework from directly learning relationships such as negation scope, coordination and discourse contrast. The framework was also specific to CAMeLBERT, Arabic hotel-review sentiment analysis and the hospitality domain, so transfer to another classifier, dialect, task or subject area would require retraining.

Its practical pipeline is demanding: every new input first requires several separate explanation methods and then token-by-token meta-model inference, adding computational cost to the explanation process. The findings therefore support a bounded conclusion rather than a general claim about explainable AI: in this annotated Arabic sentiment-analysis setting, learning which explanations resemble human judgments substantially improved plausibility while weakening faithfulness. Bridging that gap may require architectures that preserve relationships among tokens and training objectives that optimize human relevance and model fidelity together.

Subject of Research: Human-supervised aggregation of explainable AI methods for Arabic hotel-review sentiment analysis

Article Title: Learning human aligned explanations through supervised aggregation of XAI methods for Arabic sentiment analysis

Article References: Chafiqui, Y., & Anoun, H. (2026). Learning human aligned explanations through supervised aggregation of XAI methods for Arabic sentiment analysis. Discover Artificial Intelligence, 6(1), Article 1105. https://doi.org/10.1007/s44163-026-02193-0

Image Credits: AI Generated

DOI: 10.1007/s44163-026-02193-0

Keywords: Explainable AI, Arabic NLP, Sentiment analysis, Human supervision, Meta-models, CatBoost, Model faithfulness, Explanation plausibility, Token attribution, CAMeLBERT, Arabic reviews, Learning