Consistency Is Not Correctness: LLM Output Stability Fails to Predict Extraction Accuracy

Large language models asked to pull structured facts out of scientific text can repeat themselves confidently and still be wrong. That is the uncomfortable conclusion of a systematic evaluation in which four widely used generative models—GPT-4o-mini, DeepSeek-V3, the Moonshot model Kimi, and Qwen3 in non-thinking mode—were repeatedly prompted, without any fine-tuning, to extract entities and relations from a demanding plant-health dataset. The study, published in Genomics & Informatics, measured how often each model produced the same answers across repeated runs and compared that stability, called consistency, against how often the answers were actually correct. The correlation between the two turned out to be weak across every model tested.

The researchers framed their work around a practical question. Self-consistency techniques, which generate several outputs from the same prompt and pick the most frequent one, rest on the assumption that a model that agrees with itself is probably right. If that assumption held, consistency could serve as a cheap, model-agnostic filter: run the model a few times, keep only the predictions that recur, and discard the rest. Earlier work had offered encouraging hints, with one prominent study arguing that consistency is almost correctness, because stable outputs more often than not reflect accurate patterns learned from training data. But the assumption is not guaranteed—a model can be consistently and confidently wrong—and whether it holds for structured information extraction had remained largely untested.

To test it properly, the team needed a task where correctness is unambiguous and difficulty is real. They chose document-level relation extraction on the EPOP corpus, a dataset built for the epidemiomonitoring of plants. EPOP comes with a rich annotation schema: originally seven named entity types and seven binary relation types with strict constraints on which entity classes can serve as the source and target of each relation. For the experiments the authors focused on five entity types and five relation types. The corpus includes nested and discontinuous entities, synonyms, abbreviations and acronyms, and both short and long-distance overlapping relations, making it a genuinely hard benchmark rather than a toy problem.

The experimental design deliberately isolated inference-time behavior. The models were used in a hard-prompting, zero-shot setting: no fine-tuning, just a carefully crafted instruction asking each model to identify the entities in a document and return the relations between them as triplets of source, relation type, and target, formatted as JSON. Document-level extraction was chosen over text-bound extraction partly because it collapses into a single generation step, and partly because generative models struggle to pinpoint exact character offsets in text. Interestingly, asking the models to perform named entity recognition as an intermediate step before relation extraction measurably improved both the accuracy and the consistency of their relation predictions compared with asking for relations alone.

Measuring accuracy in this setting required more care than simple string matching. Because the same relation may be stated several times in a document, and because different surface forms—a species’ scientific name and its common name, for example, or a fungus and its acronym—can express the same meaning, the researchers preprocessed the annotations into document-level reference relations whose arguments carry sets of acceptable surface forms, drawn from coreference sets and entity-linking annotations to external resources such as the NCBI taxonomy and GeoNames. A predicted relation counted as correct if its type matched and its arguments matched any acceptable form. Ignoring this semantic equivalence proved costly: when accuracy was computed with strict string matching instead, scores dropped by 23 to 30 points depending on the model.

Consistency was measured with Fleiss’ Kappa, a standard inter-annotator agreement statistic generalized here so that repeated generations of the same model on the same document acted as multiple raters, and each unique relation triplet appearing across runs was treated as a subject rated present or absent in each run. Notably, this consistency measure used strict surface-form comparison and could not exploit the gold annotations, precisely because the study was testing whether consistency alone—without reference to ground truth—could stand in for accuracy. Five repetitions per document were fixed after sweeps showed that additional repetitions slightly improved consistency estimates but had minimal effect on accuracy, while higher temperature and top-p settings steadily degraded both metrics; the final experiments ran at temperature 0.2 and top-p 0.1.

The headline result is stark. At the document level, across all four models, the correlation between consistency and accuracy fell between roughly 0.10 and 0.30 in Pearson, Spearman, and Kendall’s tau tests, with DeepSeek-V3 posting the highest Spearman coefficient at about 0.30. All the correlations were statistically significant, but statistically significant and practically useful are different things: a coefficient of that size explains only a small fraction of the variation. Even when the accuracy metric was recalculated without semantic equivalence merging, the correlation barely moved, remaining weak across all tests. High consistency, in other words, does not strongly signal high accuracy in this task, and the authors conclude that consistency alone should not be relied upon as a proxy for prediction quality in relational information extraction.

The analysis also uncovered what does drive output instability. Document length, measured in bytes, and document complexity, measured as the total count of entities and relations to be predicted including duplicates, were strongly positively correlated with each other, and both were negatively correlated with model accuracy and consistency. Among the complexity components, the number of entities showed an even stronger negative association with performance than the number of relations. As documents grew longer and denser, the models both agreed with themselves less and got more wrong. Format failures added a further wrinkle: of the 825 generated answers, most malformed JSON could be repaired by post-processing, but Qwen3 produced the highest number of both recoverable and unrecoverable formatting errors, including missing property names and truncated repetitive segments, while Kimi adhered to the required format most faithfully.

One of the study’s more consequential contributions is conceptual: the distinction between recoverable variation and disruptive error. Minor typographical or formatting slips that preserve the semantic content of an extraction should not be conflated with genuine mistakes, and variations arising from synonymy or coreference—outputs that differ in wording but mean the same thing—should not penalize a model at all. The EPOP corpus’s normalization and coreference annotations allowed the team to account for many such recoverable variations in their accuracy scoring, and the substantial accuracy drop when that equivalence was removed makes the case vividly. The authors argue that evaluation metrics for generative extraction should evolve toward semantic matching, potentially drawing on coreference resolution, synonym detection, structured relation alignment, or external knowledge bases, so that semantically correct outputs are rewarded even when their surface forms differ.

The findings come with clearly stated limits. The evaluation covered a single domain, plant health, within a document-level relation extraction task, using hard prompting only; fine-tuned models could exhibit entirely different consistency–accuracy dynamics. The constrained schema counted confusions between semantically close types, such as a disease standing in for the pathogen that transmits it, as errors, and the reported correlations might not hold under looser schemas or post-processing adjustments. Consistency itself was measured by raw string comparison, which overlooks outputs that differ superficially yet remain semantically identical—a limitation that more advanced, semantics-aware consistency metrics could address, though at substantial additional inference-time cost. Within those boundaries, the message for anyone building generative extraction pipelines is direct: treat repeated-run agreement as one signal among several, not as a certificate of correctness, and invest in evaluation that understands meaning rather than merely matching strings.

Subject of Research: The relationship between output consistency and accuracy in hard-prompted large language models performing entity and relation extraction on plant-health data.

Article Title: Consistency–accuracy correlation in hard-prompted LLMs for entity and relation extraction: empirical findings from plant-health data

Article References: Yao, X., Nédellec, C., Xia, J., & Bossy, R. (2026). Consistency–accuracy correlation in hard-prompted LLMs for entity and relation extraction: empirical findings from plant-health data. Genomics & Informatics, 24(1), Article 3. https://doi.org/10.1186/s44342-025-00063-2

Image Credits: AI Generated

DOI: 10.1186/s44342-025-00063-2

Keywords: large language models, information extraction, relation extraction, consistency, accuracy, EPOP corpus, plant health, self-consistency, Fleiss Kappa, hard prompting, evaluation metrics, semantic equivalence