Tiny supervised models crush giant LLMs in fake news detection, study finds

Fake news spreads faster than corrections, and the machines we trust to catch it may not be the machines we think they are. A new benchmarking study by Abdullah Talha Kabakus of Duzce University, published in Neural Computing and Applications, delivers a sobering verdict for the era of giant language models: when it comes to detecting fabricated news without any task-specific training, big and clever is not better. In fact, some of the most celebrated open-source large language models, including reasoning-distilled systems, collapsed into a single, useless answer while a sentence embedding model with just 33 million parameters quietly outperformed them all.

The study set out to answer a question that has lingered since generative LLMs conquered headline demos: can these models spot fake news straight out of the box, with zero annotated examples or just a few, and thereby eliminate the costly labeling work that supervised systems demand? The appeal is obvious. Zero annotation cost, rapid adaptation to new domains, and the possibility of running everything on modest hardware make zero-shot and few-shot classification tantalizing for newsrooms, platforms and researchers working under tight budgets. Until now, the viability of this approach for fake news detection had remained largely uncharacterized.

To test it, Kabakus assembled nine open-source models spanning three architectural families. The first group consisted of sentence embedding models—GTE-small, BGE-small and E5-small—compact encoders trained with supervision to map text into vectors that can be compared and classified. The second group comprised standard instruction-tuned LLMs from the Llama-3.2 and Qwen2.5 families, models fine-tuned to follow commands but not specifically to judge truthfulness. The third, and perhaps most intriguing, group included reasoning-distilled models from the DeepSeek-R1-Distill line, systems engineered to work through problems step by step before answering. All were evaluated on the WELFake benchmark, a substantial dataset of 72,017 news articles.

Each model faced three prompting paradigms of increasing sophistication. Direct zero-shot prompting simply asked the model to classify an article. Chain-of-thought prompting instructed the model to reason through its answer before committing, a technique that has produced dramatic gains on mathematical and logical benchmarks. Few-shot prompting supplied three worked examples so the model could infer the task’s shape. The evaluation then measured not only standard classification metrics such as F1-score and AUC-ROC, but also label collapse susceptibility, inference throughput and the rate at which outputs could even be parsed into a valid label—a practical concern often ignored in glossy model demos.

The results were stark. GTE-small, at 33 million parameters, achieved Pareto dominance over every generative rival, scoring an F1 of 0.898 and an AUC-ROC of 0.964. The untuned generative LLMs, by contrast, managed at best an F1 of 0.667 and an AUC-ROC of just 0.326. For context, supervised fine-tuning of transformer-based models has long pushed accuracy on fake news benchmarks above 95 percent under controlled conditions. The gap between a cheap supervised encoder and an unsupervised giant was not marginal; it was categorical.

The most alarming finding lay beneath the headline numbers. Half of the evaluated generative setups—three of six—exhibited severe label collapse, a failure mode in which the model abandons genuine discrimination and defaults almost entirely to predicting the majority class, in this case labeling nearly everything as fake. Because fake and real articles are imbalanced, such a degenerate strategy can still rack up a deceptively moderate F1-score, masking the fact that the model has learned nothing useful. Only AUC-ROC, which evaluates ranking quality across the full range of decision thresholds, reliably exposed the collapse. The implication for anyone benchmarking LLMs on imbalanced classification tasks is uncomfortable: a decent-looking F1 may be hiding a model that simply says fake to everything it reads.

Would more sophisticated prompting rescue the floundering giants? No. Neither chain-of-thought reasoning nor few-shot examples resolved the label collapse, and reasoning-distilled architectures, the very models marketed for their deliberative prowess, offered no performance advantage when constrained to the rigid output formats that practical classification demands. The study also highlights prior research showing that chain-of-thought explanations can be unfaithful to a model’s actual computation, a caution that resonates here: verbosity is not verification, and elaborate reasoning traces did not translate into better truth judgments.

Efficiency compounded the embarrassment. Under the evaluation configuration used in the study, the embedding models delivered throughput advantages of 53-fold to 354-fold over the generative models. In operational terms, a lightweight encoder can screen orders of magnitude more articles per second on the same hardware, a decisive consideration for platforms confronting millions of posts daily or for edge deployments on resource-constrained devices. Scale, in this comparison, bought slowness and miscalibration rather than insight.

The study is careful about its own scope. It proposes no new architecture and introduces no new dataset; its contribution is a systematic empirical characterization, one of the first evaluations of reasoning-distilled open-source models on this task, and a multi-metric evaluation protocol designed specifically to diagnose probability miscalibration in generative models under resource-constrained deployment. The findings are explicitly tied to the evaluated experimental conditions: on the WELFake benchmark, with these nine models and these three prompting strategies, scale and prompting sophistication could not substitute for supervision.

The broader lesson extends well beyond fake news. As institutions rush to replace trained classifiers with prompt-based LLM pipelines, this work stands as a quantitative warning that generative fluency does not guarantee discriminative competence, and that evaluation protocols must be robust enough to catch degenerate behavior before deployment. The author has released full algorithmic details, hyperparameters and verbatim prompt templates, and all models and the WELFake dataset are publicly available, inviting replication and extension. In the contest between a 33-million-parameter workhorse and the giants of open-source AI, the workhorse won—cheaply, quickly, and honestly.

Subject of Research: Benchmarking open-source LLMs and sentence embedding models for zero-shot and few-shot fake news detection

Article Title: Scale is not a substitute for supervision: benchmarking open-source LLMs and embedding models for fake news detection

Article References: Kabakus, A. T. (2026). Scale is not a substitute for supervision: benchmarking open-source LLMs and embedding models for fake news detection. Neural Computing and Applications, 38(20), Article 808. https://doi.org/10.1007/s00521-026-12557-3

Image Credits: AI Generated

DOI: 10.1007/s00521-026-12557-3

Keywords: fake news detection, large language models, sentence embeddings, zero-shot classification, chain-of-thought prompting, DeepSeek-R1, Llama-3.2, Qwen2.5, WELFake dataset, label collapse, AUC-ROC, model benchmarking