ModernBert outperformed larger language models when generalizing to unseen social media data for depression detection, achieving the top accuracy score on an independent test set despite larger architectures dominating familiar validation benchmarks. In a study published in the International Journal of Computational Intelligence Systems, researchers conducted a comprehensive benchmark to evaluate the robustness of diverse language model architectures under domain shifts, text cleaning variations, and differences in training data volume.
Addressing the growing volume of social media text that provides critical insights into public mental health, the team set out to explore scalable and reliable artificial intelligence tools for early depression detection that can withstand the complex variations encountered in real-world environments.
To examine resilience across different model families, the researchers evaluated six prominent language model architectures: LLaMA-1b, LLaMA-3b, ModernBert, Bert, GPT-2, and Pythia-1b. The candidate models were systematically trained on subsets of the Sentiment140 dataset, comparing samples of 50,000 rows against larger allocations of 100,000 rows. The experimental setup also tested each dataset size with and without text preprocessing to determine whether an aggressive text cleaning pipeline improves classification outcomes.
To test generalization beyond the initial training distribution, all trained models were subjected to evaluation on an independent, unseen, depression-focused benchmark known as the DD-Kaggle dataset. The researchers assessed performance across multiple standardized classification metrics, measuring accuracy, precision, recall, F1-score, and area under the receiver operating characteristic curve. This evaluation protocol separated performance on in-distribution validation data from true model robustness when facing completely unencountered mental health text.
On the in-distribution validation sets, a clear performance hierarchy materialized across the competing architectures. The larger LLaMA-3B model demonstrated superior capability within familiar data conditions, achieving a peak validation accuracy of 92.33 percent on the 50,000-row uncleaned dataset. However, this high validation performance did not ensure superior generalization to unfamiliar texts, illustrating that in-distribution benchmarks do not always reflect real-world operational effectiveness.
When tested on the unseen out-of-distribution DD-Kaggle dataset, ModernBert achieved the highest overall accuracy, securing a top score of 72.42 percent. Every evaluated model experienced a significant performance drop when transitioning from in-distribution validation data to the independent test set, underscoring domain shift as a primary obstacle for automated mental health classification. Despite its smaller scale relative to LLaMA-3B, ModernBert demonstrated greater architectural suitability for generalizing across distinct domains.
The experimental results also revealed distinct impacts from training scale and text cleaning protocols. Increasing the training data volume from 50,000 to 100,000 rows consistently enhanced model performance across the board. In contrast, applying the aggressive preprocessing pipeline evaluated in the study produced mixed and model-dependent effects, demonstrating no consistent performance improvement from text cleaning across the different architectures.
The authors concluded that evaluating language models on independent, unseen test sets is essential for determining their true operational robustness in mental health monitoring. While larger models excel on in-distribution data, architectural design choices such as those in ModernBert represent a critical factor in achieving effective domain generalization. The researchers emphasized that rigorous testing accounting for domain shift remains paramount for developing dependable artificial intelligence systems in mental health applications.
Subject of Research: Language model robustness for depression detection in social media text
Article Title: ModernBert to LLaMA: a Benchmark Study of LLM Robustness for Depression Detection in Social Media Texts
Article References: Narayan, Y. P., Dubey, A. K., Jain, A., Panwar, A., Varghese, A., Krejcar, O., Herout, L., & Namazi, H. (2026). ModernBert to LLaMA: a Benchmark Study of LLM Robustness for Depression Detection in Social Media Texts. International Journal of Computational Intelligence Systems. https://doi.org/10.1007/s44196-026-01636-x
Image Credits: AI Generated
DOI: 10.1007/s44196-026-01636-x
Keywords: depression detection, language models, ModernBert, LLaMA-3b, LLaMA-1b, Bert, GPT-2, Pythia-1b, mental health monitoring, social media text, Sentiment140, DD-Kaggle, domain shift, out-of-distribution generalization, text preprocessing, model robustness
News Source: Glenn Wilkins. (November 19, 2026). ModernBert Outperforms Larger Models in Generalizing Social Media Depression Detection. Scienmag.



