LLM study of 67 abstracts finds strong local QA, weak global consistency

A large language model pipeline designed to turn scientific abstracts into factual and counterfactual question-answer pairs achieved high automated scores for individual components but performed poorly when asked to maintain coherence across an entire document. Among 67 abstracts with complete generated artifacts, GPT-5-mini scored original question answering at 0.97, relation validity at 0.89 and counterfactual question answering at 0.85. Yet its document-wide consistency score was only 0.07. The study therefore found a pronounced gap between artifacts that appeared valid in isolation and a pipeline output in which all extracted objects, relationships and hypothetical changes formed a mutually consistent account.

The research was an evaluation of an implemented five-stage generation-and-validation pipeline, not a foundation-model training study. GPT-4.1, used with a temperature of 0.0 and a maximum output of 1,200 tokens per call, first extracted entities, events and explicitly stated causal or dependency relations from an abstract. It then compressed that structure into a lightweight structural causal model, or SCM, containing broad variables such as method, data, intervention, outcome and metric. The system next generated five factual question-answer pairs for each abstract, produced intervention-conditioned counterfactual counterparts, and filtered pairs that were logically inconsistent, unanswerable or incompatible with the stated change. No foundation model was trained or fine-tuned, and the SCM served as a prompt-level intervention scaffold rather than a method for discovering true causal relationships from the abstracts.

The source material was a heterogeneous Scopus CSV export containing 100 records with titles and abstracts. Its subjects included medical artificial intelligence, industrial and manufacturing applications, robotics, and human-robot collaboration. The notebook processed records in file order rather than using domain-stratified sampling. A document entered the validation cohort only when the pipeline generated and stored the complete set of artifacts required for evaluation. Sixty-seven documents met that requirement, creating a complete-artifact cohort rather than a random 67-document sample; the other 33 records were not included because their outputs were incomplete. Each successfully processed abstract contributed five factual question-answer anchors, for 335 factual pairs in the validation cohort before counterfactual generation and filtering.

Counterfactual generation was intended to produce paired examples under controlled changes rather than arbitrary paraphrases. For each factual question, the model was instructed to select one intervenable scientific variable, assign it a plausible alternative value, preserve the question’s form as closely as possible and update the answer consistently with the intervention. An example drawn from an abstract evaluating ChatGPT, Copilot and Perplexity asked which three language models had been assessed. Its counterfactual changed the model set to ChatGPT, Gemini and Llama 2, recorded that method intervention explicitly and updated the answer accordingly. Other examples could change a dataset or cohort size, or alter the quality attributed to generated explanations and revise the associated study conclusion. These examples illustrate the pipeline’s intended transformation semantics, not proof that the generated alternatives represent the unique real-world consequences of those changes.

The final filtering stage checked individual counterfactual pairs for logical consistency, answerability and compatibility with the intervention description. It did not perform an iterative, document-wide reconciliation across the extracted entities and events, the SCM, the factual questions and every counterfactual question. That design choice was central to the observed results. A locally acceptable answer could remain in the output even when an earlier omission or reinterpretation conflicted with another part of the document. The authors identified three plausible design-level contributors: independent model calls at each stage, compression of scientific content into a small set of broad SCM variables, and a filter centered on pair-level validity. Because the experiments did not include matched no-SCM, no-filter or cross-stage-reconciliation conditions, these factors were plausible explanations rather than individually established causes.

Validation combined automated scoring with independent human assessments. A separate GPT-5-mini invocation received each source abstract, the extracted entities and events, the induced SCM, factual questions, pre-filter and filtered counterfactual questions, and a log excerpt. Under a strict JSON schema, it assigned a binary score and concise evidence-based comment for each of 15 criteria covering entities, events, relations, original question answering, counterfactual question answering and global consistency. The validator was instructed to use the abstract as primary evidence and assign zero when evidence was unclear. Because both generation and automated validation used OpenAI models, this was a distinct model invocation but not independent validation from another model family.

Three human annotators independently scored the same 67 documents on all 15 criteria, yielding 1,005 item-criterion decisions per person, or 3,015 decisions in total. The released protocol did not record formal domain-specific qualifications for the annotators and included no adjudicated gold standard. Their scores therefore measure agreement among the participating raters rather than agreement with an expert consensus. Inter-rater reliability was moderate: mean pairwise Cohen’s kappa was approximately 0.54, while the three reported pair values were about 0.68, 0.54 and 0.38. Agreement was more stable for checks that could be made directly against the source, such as whether an entity appeared explicitly or an answer was factually correct. It was more variable for judgments about omissions, event completeness, counterfactual acceptability and end-to-end coherence.

The automated validator showed the same broad division between local grounding and holistic interpretation. Original question answering received a mean score of 0.97, relations 0.89 and counterfactual question answering 0.85. Entities scored 0.68 and events 0.60. The global-consistency criterion, which assessed whether the entire pipeline output was internally consistent, received only 0.07. Criterion-level results showed particularly high pass rates around 0.97 for several original and counterfactual QA checks, while entity-completeness and event-completeness criteria were lower. The magnitude of the difference between Q10–Q14 and Q15 shows why passing checks on individual questions cannot establish that a complete abstract-derived package is coherent. It does not establish the severity of any individual error or show that every contradiction would be missed when artifacts were reviewed separately.

The researchers also measured intervention sensitivity, defined as the absolute difference between normalized factual and counterfactual validation scores. They introduced the intervention degradation rate, or IDR, for documents whose counterfactual score fell beyond a predefined threshold for a meaningful decrease. Seventeen of the 67 documents, or 25.4%, crossed that threshold, while 50 documents, or 74.6%, did not. These are descriptive stability statistics for changes to scientific-content variables such as methods, datasets, evaluation settings, outcomes and metrics. They are not measures of demographic bias, protected-attribute bias propagation or counterfactual fairness. Moreover, both scores came from the same validation framework, so observed displacement can reflect generation errors, evaluator sensitivity or a combination of the two.

Within these limits, the study supports criterion-level auditing while warning against equating local validity with document-wide reliability. The experiments characterize one end-to-end pipeline and do not show that the SCM or filtering stage outperforms direct factual prompting, unfiltered generation, non-causal rewriting or a retrieval-augmented system. Retrieval-specific measures were not applicable because each target abstract was supplied directly instead of retrieved from a larger corpus. The authors identify cross-stage reconciliation as a possible next step: a mechanism could compare entities, events, SCM edges, factual answers and counterfactual consequences, then return detected contradictions to the relevant generation stage. That extension was motivated by the failure pattern but was not tested here. The supported conclusion is therefore narrow but important: locally credible scientific questions and counterfactual pairs can be produced under controlled prompts, yet the complete pipeline may still have difficulty making the extracted scientific scenario stand together as one coherent account.

Subject of Research: Evaluation of an LLM pipeline for generating and validating factual and counterfactual question-answer pairs from scientific abstracts

Article Title: Structurally conditioned counterfactual generation and validation for question answering from scientific text

Article References: Malashin, I. P., Martysyuk, D. A., Tynchenko, V. S., Gantimurov, A. P., Nelyub, V. A., & Borodulin, A. S. (2026). Structurally conditioned counterfactual generation and validation for question answering from scientific text. Frontiers in Computer Science, 8, Article 1915137. https://doi.org/10.3389/fcomp.2026.1915137

Image Credits: AI Generated

DOI: 10.3389/fcomp.2026.1915137

Keywords: large language models, scientific text, question answering, counterfactual generation, structural causal models, automated validation, human annotation, document consistency, scientific abstracts, Structurally, conditioned, counterfactual