AI Models Trained on Old Rumour Data Falter on Irish Rumours, New Dataset Shows

Automatic systems that judge whether social media replies support, reject or question a rumour perform far worse when they are moved from the datasets they were trained on to a new domain, according to a peer-reviewed study published in EPJ Data Science on 11 September 2026. The paper, authored by Yue Li, Cian McGrath, Kirsty Park, Eileen Culloty, Kalina Bontcheva and Carolina Scarton, introduces a new dataset called ISLES, which focuses on rumours that spread online in Ireland. The authors report that top-performing models trained on the widely used RumourEval data suffer considerable performance degradation when evaluated on ISLES, and they present a self-training framework that uses large language models to generate synthetic Irish-domain training data, boosting the performance of a RumourEval-trained model by nearly 127 percent on the new dataset.

Rumour stance classification is a task in which a conversation begins with a rumour-introducing source post on social media, and the system must identify the opinion of its replies towards that post. The categories used in the study are confirming, where the author of a response supports the source tweet; rejecting, where the author disagrees with it; questioning, where the author asks for additional evidence in relation to it; and commenting, where the author makes their own remark without a clear stance. The researchers note that sarcasm complicates annotation: a reply that sarcastically or humorously supports a source tweet should be treated as rejecting, while a sarcastic or humorous denial should be treated as confirming. They also caution that the appearance of question marks does not necessarily indicate a questioning stance, since rhetorical questions, which do not expect an answer, should not be coded as questioning.

The motivation for the new dataset comes from a gap the authors identify in the field. Although methods for automatic rumour stance classification have yielded high performances, none of them had been evaluated on a dataset that is substantially different from the data they were trained and tested on, which, the paper argues, puts their reliability in real-world applications into question. ISLES is designed to fill that gap by featuring substantial differences from commonly used datasets over topic, lexicon and stance distribution. The dataset covers online rumours spread in Ireland, and the annotation process required annotators to infer the stance of replies towards source tweets, with conversation threads split into triples of source tweet, parent tweet and target tweet for indirect replies, and tuples of source tweet and target tweet for direct replies.

When the researchers evaluated the top-performing models trained with RumourEval data on ISLES, they observed considerable performance degradation compared to the performance on the RumourEval test sets. Among the models examined were BERTweet-large and RoBERTa-large, fine-tuned with a class-weighted loss function whose class weights were computed according to the class distribution of the RumourEval 2019 training data. Hyperparameters were tuned by grid search, with the optimal settings determined based on the weighted F2 score on the validation set, and each fine-tuning process was repeated five times with different random seeds. The team also tested a hierarchical BERT implementation for rumour stance detection, retrained on the RumourEval 2019 training set, and compared these against large language models used in a zero-shot in-context learning setting.

For the zero-shot experiments, the researchers tested twelve different prompts with Llama and Mistral models and chose the optimal one based on its performance over the RumourEval validation set. The prompts phrased the instruction in various ways, using different words to denote the stance classes, inspired by the stance names used in other stance classification datasets, such as in favour and against. Because the models occasionally produced output that did not match any of the expected stance names, the team adopted a regular-expression post-processing step to extract the stance class, and in the very few cases where no stance name was matched, they assumed the reply belonged to the comment class, since that is the majority class of the RumourEval dataset.

The central contribution of the paper is a self-training framework for domain adaptation that works in the absence of both labelled RumourEval and Irish-domain rumour stance data. The framework uses large language models to generate synthetic Irish-domain rumour stance data, which is then used to adapt a RumourEval-trained model to the domain of rumours spread in Ireland. Using synthetic data generated with Llama 2 (13b), the proposed framework successfully boosts the performance of the RumourEval-trained model by nearly 127 percent on ISLES, reaching a weighted F2 score of 0.5719. By comparison, zero-shot in-context learning with the same Llama 2 (13b) model achieved a weighted F2 score of 0.4312 on the same dataset.

The authors also examined whether the approach generalises to other generative models. They repeated the synthetic data generation procedure with Llama 3 (8b) and DeepSeek (7b), and found that both models were able to leverage target-domain knowledge to generate diverse replies expressing the intended stance labels. When the synthetic data produced by these models was used for adaptation, the framework demonstrated substantial improvements, with synthetic data from Llama 3 (8B) and DeepSeek (7B) improving the performance of the RumourEval-trained model on ISLES by 131 percent and 133 percent, respectively. However, generating natural and human-like query and comment replies remained challenging for these more recent models, particularly for the comment class. When prompted to generate comments, Llama 3 (8b) and DeepSeek (7b) often started by implicitly agreeing with the source tweet and then raising a related question unrelated to assessing the veracity of the rumour.

The team subjected their evaluation to several robustness checks. A nonparametric bootstrap analysis with 10,000 resamples of ISLES produced an average weighted F2 of 0.5705 for the proposed method, with a 95 percent bootstrap confidence interval running from 0.5463 to 0.5983, which the authors say suggests the evaluation metric is reasonably stable under repeated resampling. They also varied the high-confidence threshold used in the framework from 0.9 to 0.95 and found that although their empirical choice of 0.9 was not optimal, performance remained relatively stable under moderate threshold variations, indicating that the framework is relatively robust to that choice. An error analysis showed that the model could relatively generalise well on samples where the stance is expressed explicitly, but when support or deny stances are conveyed through elaboration on target-domain knowledge, the model tends to predict them as the comment class.

The researchers also weighed the computational cost of their approach. Estimating that a decoder-only large language model with N parameters uses 2N FLOPs per token for inference, they calculated that the one-time cost of synthetic data generation in their framework is approximately equivalent to performing zero-shot in-context learning inference on 572 tweets from ISLES, corresponding to roughly 50 percent of the dataset. Importantly, the large language model is used only once during the adaptation framework to generate synthetic data; all subsequent training and inference are performed using BERTweet-large, which is substantially more computationally efficient than LLM-based inference. Given that the adaptation framework achieves a weighted F2 score of 0.5719 compared with 0.4312 for zero-shot in-context learning using the same Llama 2 (13b) model, the authors conclude that the one-time generation cost is justified by the resulting performance gains and the reduced inference cost of the final adapted smaller BERTweet model.

The study is published as open access, and the version shared at publication is an accepted, citable manuscript carrying a permanent DOI, subject to further edits before the final Version of Record. Its practical message is directed at the rumour-detection research community: models that look strong on the benchmarks they were trained for cannot be assumed to transfer to new domains, and the topic, lexicon and stance distribution of a target domain can differ enough from standard training data to erode performance considerably. The ISLES dataset and the synthetic-data adaptation framework offer a way to measure that gap and, in the absence of labelled target-domain data, to narrow it, while the reported error patterns indicate that stances expressed through domain-specific elaboration remain a difficult case for adapted models.

Subject of Research: Domain adaptation of rumour stance classification models to Irish online rumours using a new dataset and LLM-generated synthetic data.

Article Title: Rumour stance classification adaptation under domain shift: a case study of rumours in Ireland

Article References: Li, Y., McGrath, C., Park, K., Culloty, E., Bontcheva, K., & Scarton, C. (2026). Rumour stance classification adaptation under domain shift: a case study of rumours in Ireland. EPJ Data Science. https://doi.org/10.1140/epjds/s13688-026-00696-5

Image Credits: AI Generated

DOI: 10.1140/epjds/s13688-026-00696-5

Keywords: rumour stance classification, domain adaptation, ISLES dataset, RumourEval, large language models, synthetic data, self-training, social media rumours, Ireland, BERTweet, natural language processing, misinformation