AI Fact-Checks Read Better Than They Argue, and Readers Prefer Them Anyway

Large language models write COVID-19 fact-checks that readers like more than the human originals, even though the AI versions score worse on nearly every traditional measure of health communication quality. That paradox is the central finding of a new case study published in BMC Artificial Intelligence, which compared the communication styles of LLM-generated and human-written explanations of false health claims and then asked ordinary readers which they preferred.

The claims came from two peer-reviewed datasets spanning February to July 2020, with veracity determined by manual annotation rather than algorithms. Using chain-of-thought prompting built on journalism information assessment guidelines, the researchers generated matching AI fact-checking articles with three state-of-the-art models: GPT-4, Llama-2-70B and Mistral-7B, each run in zero-shot and few-shot variations.

The prompting strategy instructed the models to classify each claim’s veracity and then justify that classification by considering source credibility, the availability of tested evidence, and possible alternative explanations. Despite an explicit request for articles of around 800 words, the LLMs consistently produced shorter texts, averaging 335 words against an average of 765 words for the human articles. The final corpus contained 1,498 AI articles in each prompting variation, randomly balanced across the three models.

To compare styles, the researchers operationalized communication along three components drawn from health communication theory: the information itself, the sender, and the receiver. Linguistic features included certainty, measured with a model developed by Pei and Jurgens; cognitive processing attributes, measured with the LIWC psycholinguistic lexicon; and readability, measured with the Automated Readability Index and Flesch-Kincaid grade level. Persuasive strategies of credibility, evidence and impact were scored with a hierarchical weakly-supervised model, while value alignment was assessed through Schwartz social values and moral foundations such as authority and fairness. Because the AI articles were shorter, the team used paragraph-wise scoring for transformer-based measures and length-normalized count-based measures to reduce bias.

The analysis of variance showed significant differences on every communication style dimension. The LLM articles scored significantly lower on persuasive strategies, with less language impact and less evidence use than the human-authored pieces. They expressed significantly less certainty, although the mean difference on that dimension was smaller than for the others. They also scored lower on moral foundations of authority and fairness, and showed weaker alignment with the social values of power, achievement, tradition and conformity. On the cognitive side, the AI texts contained significantly more cognitive processing expressions, suggesting more explicit reasoning in the writing itself.

Readability produced a counterintuitive result. The LLM articles had significantly higher readability scores, which in these metrics means more complex and harder to read, not easier. The authors note this could be especially problematic for misinformation corrections, since harder-to-read text may hinder understanding across a diverse audience and weaken trust. Yet this apparent deficit did not translate into reader rejection.

For the human evaluation, the researchers recruited 99 participants from the crowdsourcing platform Prolific, all adults living in America who could read English. Participants were told the study examined different writing strategies for fact-checking responses but were not informed that some articles were AI-generated. To control for length, the team built 167 article pairs in the 250-to-400-word range where human and AI versions had closely matched average and median lengths. Each participant rated five randomly selected pairs, with presentation order shuffled, answering three questions on seven-point scales: which article used clearer language, which provided all the necessary information, and which was more persuasive.

The preference results were striking. Out of 495 total responses, 308 (62.23 percent) preferred the LLM articles for clarity of language, 307 (62 percent) preferred them for providing necessary information, and 306 (61.8 percent) found them more persuasive. A Wilcoxon Signed-Rank Test rejected the null hypothesis of no preference, confirming a significant lean toward the AI content. Regression analyses using cumulative link mixed models found that male participants tended to prefer LLM fact-checking more, and that AI articles easier to read were rated as clearer. Interestingly, when LLM articles used fewer authority-related expressions than their paired human articles, participants were actually more likely to find the AI versions persuasive, a result that runs against prior literature linking authority and source credibility to persuasiveness.

Qualitative analysis of participants’ explanations revealed five drivers of the AI preference: focused presentation that led with the main point, an objective and fact-based standpoint, a professional vibe including references from reputable sources, accessible language using everyday terms, and a calm, non-emotional tone. Some human-written articles, by contrast, were perceived as scattered, opinionated, or even angry and clickbait-like. Notably, the researchers observed that the LLMs’ tendency to cite reputable sources, even when those citations were incorrect or non-existent, contributed to a perception of professionalism and evidence support.

The authors caution that these preferences should be interpreted only at the level of communicative appeal rather than factual reliability, since the study did not evaluate the factual quality of the AI outputs. The study is situated in the U.S. context of COVID-19 fact-checking, treats professional fact-checking articles as a relative rather than absolute reference standard, and did not assess qualities such as coherence or informativeness. Still, the conclusion is pointed: readers may prioritize structured, neutral, professional-looking presentation over the nuanced reasoning, moral alignment and holistic coverage that experts associate with high-quality health communication. The researchers suggest LLMs could complement rather than replace human fact-checkers, but warn that their weaker evidence engagement and value alignment pose real risks in high-stakes health settings, underscoring the need for human oversight and careful evaluation before deployment.

Subject of Research: Communication styles and reader preferences of LLM-generated versus human-written COVID-19 fact-checking explanations

Article Title: Communication styles and reader preferences of LLM- and human-authored COVID-19 information explanations: a case study

Article References: Zhou, J., Venkatachalam, K., Choi, M., Saha, K., & De Choudhury, M. (2026). Communication styles and reader preferences of LLM- and human-authored COVID-19 information explanations: a case study. BMC Artificial Intelligence, 2(1), Article 10. https://doi.org/10.1186/s44398-026-00030-z

Image Credits: AI Generated

DOI: 10.1186/s44398-026-00030-z

Keywords: large language models, fact-checking, health misinformation, COVID-19, health communication, GPT-4, reader preferences, communication styles, moral foundations, persuasion, readability, AI-generated text