Arabic retrieval-augmented generation tested across 198 configurations finds best system reaches 89.29% accuracy

Retrieval-augmented generation (RAG) has become a widely used approach for improving factual question answering with large language models, but its design for Arabic has remained poorly understood, especially in specialized domains that demand both linguistic precision and domain expertise. A new study published in Neural Computing and Applications addresses this gap with a systematic evaluation of Arabic RAG architectures applied to Maliki Islamic jurisprudence, a field requiring deep legal and religious knowledge.

The researchers evaluated 198 RAG configurations, combining 11 embedding models, 6 neural re-ranking strategies, and 3 large language models: Allam-7B, Fanar-9B, and Llama3-8B. The systems were tested on a benchmark of 140 expert-curated multiple-choice questions. The best-performing configuration combined CAMeLBERT, a lightweight Arabic neural re-ranker, and Allam-7B, achieving 89.29% accuracy and substantially outperforming standalone generation without retrieval.

Statistical analyses, including Cochran’s Q, McNemar’s test, and the Holm–Bonferroni correction, showed that the benefit of retrieval refinement varies substantially across generators. The improvement was largest for the multilingual model, more modest for one Arabic-specialized model, and smallest for another Arabic-specialized model with comparable standalone accuracy. According to the authors, this indicates that intrinsic Arabic competence alone does not fully determine how much a generator benefits from retrieval.

To explain these results, two domain experts independently assessed 800 retrieval instances, evaluating retrieval relevance, retrieval sufficiency, and error attribution. The qualitative analysis confirmed that Arabic-specific retrieval provides more relevant supporting evidence, and revealed that as retrieval improves, residual errors increasingly reflect limitations at the generation stage rather than retrieval quality.

From these findings the study proposes a concept the authors call Capacity–Retrieval Alignment, which characterizes retrieval effectiveness as emerging from the interaction between retrieval quality, generator capacity, and the generator’s capacity to utilize retrieved evidence. The framework is intended to provide a principled basis for designing Arabic retrieval-augmented systems.

The study, authored by Hassan Ben Ayed, Ouissem Ben Fredj, and Omar Cheikhrouhou, was published online on 27 October 2026 in Volume 38 of the journal as article number 852.

The finding that generation-stage limitations dominate once retrieval quality is high points to generator capacity as a key bottleneck for further progress in specialized Arabic domains.

The authors note that the evaluation was conducted on a single specialized domain and a fixed benchmark of 140 questions, so the reported accuracies and the proposed alignment framework should be interpreted within that scope. The full article is available via subscription access from Springer.

Subject of Research: Evaluation of Arabic embedding models, re-ranking strategies, and large language models in retrieval-augmented generation for Maliki Islamic jurisprudence question answering.

Article Title: Evaluating Arabic embeddings in retrieval-augmented generation: a comprehensive multi-LLM study with advanced re-ranking strategies

Article References: Ayed, H. B., Fredj, O. B., & Cheikhrouhou, O. (2026). Evaluating Arabic embeddings in retrieval-augmented generation: a comprehensive multi-LLM study with advanced re-ranking strategies. Neural Computing and Applications, 38(20), Article 852. https://doi.org/10.1007/s00521-026-12596-w

Image Credits: AI Generated

DOI: 10.1007/s00521-026-12596-w

Keywords: retrieval-augmented generation, Arabic NLP, large language models, embedding models, neural re-ranking, Maliki Islamic jurisprudence, Allam-7B, Fanar-9B, Llama3-8B, CAMeLBERT, question answering, Capacity-Retrieval Alignment