A comprehensive comparative evaluation of similarity metrics within Chinese retrieval-augmented generation systems demonstrates that character-level lexical matching can rival dedicated semantic embedding models on extractive question answering tasks. The investigation evaluated eight distinct distance and similarity metrics alongside specialized Chinese text encoders across full development benchmarks. The results reveal that when natural-language queries share verbatim textual anchors with underlying reference passages, windowed character-level edit distance achieves downstream generation quality that is statistically indistinguishable from leading contrastively trained Chinese dense retrievers. However, the study also reveals that this equivalence depends entirely on the query structure, as the performance of surface-matching algorithms degrades sharply when tested against concise keyword-style search queries.
Retrieval-augmented generation has become a primary architectural strategy for mitigating factual hallucinations in large language models by grounding generative responses in retrieved external evidence. While previous academic studies have predominantly evaluated dense vector retrieval using cosine similarity over English benchmarks, systematic investigations comparing alternative mathematical metrics within Chinese language contexts have remained scarce. Chinese text presents distinct linguistic properties, such as a lack of morphological inflection and reliance on character sequences rather than explicit word boundaries. The researchers sought to determine whether computationally intensive neural embedding infrastructure is universally necessary for evidence retrieval in Chinese language tasks or whether simpler token-level and character-level mathematical formulations could offer practical and competitive alternatives.
To investigate these dynamics, the study implemented an end-to-end retrieval, generation, and statistical evaluation framework spanning two separate Chinese datasets and two deterministically configured open-source language models. The primary evaluation utilized the complete development set of CMRC2018, an extractive machine reading comprehension dataset comprising 3,219 span-annotated question-answer pairs and a preprocessed knowledge corpus of 2,026 passages. For external validation, the authors introduced the official development split of DuReader-robust, containing 1,417 questions derived from real-world search engine queries and 2,062 processed passages. The experimental pipeline compared traditional word-level Jaccard similarity, Weighted Jaccard, Dice coefficient, character n-gram Jaccard, and multiple formulations of Levenshtein edit distance against dense vector representations, including the multilingual Sentence-BERT MiniLM baseline as well as two high-performing Chinese contrastive encoders, bge-base-zh-v1.5 and m3e-base.
The experimental execution coupled upstream passage retrieval with deterministic answer generation on consumer-grade computing hardware. Downstream generation was conducted using DeepSeek-R1-Distill-Qwen-14B alongside cross-generator validation using Qwen2.5-7B-Instruct, both invoked under strict greedy decoding parameters with zero temperature and fixed random seeds to guarantee byte-identical reproducibility. In the retrieval phase, the authors operationalized retrieval recall as answer-containment recall, marking a hit whenever a normalized gold reference string appeared as a substring within the top-k retrieved passages. For downstream generative quality, the evaluation tracked Exact Match and character-level F1 scores, as well as the unsupported rate—defined as the proportion of generated responses that could not be located verbatim within any retrieved evidence passage, serving as a strict proxy for hallucination.
Initial evaluations on the full CMRC2018 development set established that strong contrastive Chinese embeddings achieved the highest retrieval recall, with bge reaching 95.34 percent and m3e attaining 95.28 percent at a retrieval depth of four passages. Crucially, the researchers identified and resolved a critical implementation artifact in traditional edit-distance retrieval. Naive implementations compute the raw edit distance between a brief query and an entire passage, which inherently penalizes longer text spans and erroneously retrieves the shortest available passages regardless of topical relevance, causing retrieval recall to collapse to 1.03 percent. By implementing a sliding-window character partial-match formulation that aligned the query against the best-matching substring of each passage, the corrected windowed Levenshtein metric attained 93.79 percent retrieval recall, placing it firmly in the top tier alongside the neural encoders.
Downstream generative performance on the complete benchmark mirrored these retrieval dynamics while demonstrating that raw recall differences do not necessarily translate into generation disparities. Against the multilingual Sentence-BERT cosine baseline, which achieved an F1 score of 46.08 and an Exact Match rate of 21.12 percent, the leading Chinese embedding bge gained 19.4 F1 points to reach 65.5 points, while the corrected windowed Levenshtein metric gained 19.3 F1 points to reach 65.4 points. Formal statistical testing using paired bootstrap confidence intervals and McNemar exact tests with Holm–Bonferroni correction confirmed that the differences in F1, Exact Match, and passage support between bge and windowed Levenshtein were statistically indistinguishable. Both top-tier methods also reduced the unsupported generation rate by more than 11 percentage points compared to the multilingual baseline, demonstrating that lexical surface matching can ground generative responses effectively without neural encoders.
To evaluate whether empirical findings derived from small evaluation sets remain robust, the authors conducted an extensive resampling stability analysis. By drawing 1,000 random subsamples at various sample sizes from the full development set results, the study quantified the ranking flip rate—the statistical probability that a random subsample would identify an incorrect method as the top performer. At an evaluation size of ten queries, the apparent winner differed from the true full-set winner in 37.6 percent of draws, and this instability persisted at 14.6 percent for sample sizes of one hundred queries. The ranking flip rate only dropped to zero once sample sizes reached one thousand instances. These findings emphasize that small-sample evaluations in retrieval-augmented generation research carry a substantial risk of misidentifying optimal configurations due to high sampling variance.
The investigation subsequently tested the external validity of these findings by evaluating all retrieval methods on the DuReader-robust dataset, where an important operational boundary emerged. Unlike CMRC2018, which features syntactically complete natural-language questions, DuReader-robust consists of concise search-engine keyword queries averaging only 9.4 characters in length. Under these conditions, the character-level lexical metrics suffered severe performance drops: windowed Levenshtein retrieval recall fell from 93.79 percent to 67.61 percent, and character n-gram Jaccard declined from 87.36 percent to 55.82 percent. Conversely, the dense Chinese embedding models maintained high retrieval recall, with bge reaching 98.31 percent and m3e reaching 97.46 percent. Subsequent end-to-end generation confirmed that bge outperformed windowed Levenshtein by 11.5 F1 points on DuReader-robust, demonstrating that lexical retrieval cannot substitute for semantic representations when queries lack verbatim anchors.
Cross-generator validation further demonstrated that while absolute performance metrics depend heavily on the underlying language model, relative tier rankings among retrieval methods remain stable. When substituting the 14-billion-parameter reasoning model with the 7-billion-parameter instruct model on identical question subsets, average F1 scores rose from 62.5 to 80.9 because the instruct model provided concise answers directly without exhausting context tokens on extended chain-of-thought reasoning segments. Across both generators, the top-tier retrievers consistently maintained statistically significant advantages over the multilingual embedding baseline. Additionally, single-threaded computational profiling on consumer hardware revealed that windowed character matching operates without GPU requirements or persistent model weights, requiring 17 to 22 milliseconds per query, compared to 8 to 11 milliseconds of GPU encoding latency and 409 megabytes of memory storage for the best embedding encoder.
The study highlights specific constraints and boundaries in its experimental scope, noting that evaluations were conducted exclusively on extractive Chinese question answering benchmarks where answers correspond directly to text spans. The researchers emphasized that the unsupported rate measures verbatim text containment rather than semantic entailment, treating valid abstractive paraphrases as unsupported content. Furthermore, asymptotic computational profiling indicated that while character matching scales linearly with corpus size, dense retrieval using approximate nearest-neighbor indexing will maintain clear scalability advantages in massive document collections. Ultimately, the researchers established a practical decision framework for Chinese system deployment: specialized contrastive embeddings provide the most reliable default across varied query types, while corrected character-level metrics offer an efficient zero-infrastructure alternative exclusively when user queries reliably preserve verbatim lexical anchors.
Subject of Research: Comparative evaluation of similarity metrics in Chinese retrieval-augmented generation systems
Article Title: Comparative evaluation of similarity metrics for Chinese retrieval augmented generation
Article References: Chen, R. (2026). Comparative evaluation of similarity metrics for Chinese retrieval augmented generation. Discover Artificial Intelligence, 6(1), Article 1675. https://doi.org/10.1007/s44163-026-02613-1
Image Credits: AI Generated
DOI: 10.1007/s44163-026-02613-1
Keywords: Retrieval-Augmented Generation, Large Language Models, Natural Language Processing, Chinese Information Retrieval, Similarity Metrics, Text Embeddings, Levenshtein Distance, Jaccard Similarity, CMRC2018, DuReader-robust, Hallucination Mitigation, Unsupported Rate, Sentence-BERT, Exact Match, F1 Score, DeepSeek-R1, Qwen2.5, Consumer Hardware Benchmarking
News Source: Denise Maddox. (November 14, 2026). Chinese RAG Benchmarks Reveal Simple Lexical Metrics Can Match Vector Embeddings. Scienmag.



