Quantification Models Reveal Key Behavioral Predictors Behind Online Content Moderation Responses

A large-scale computational study analyzing the digital footprints of 16,828 social media users affected by Reddit’s major 2020 deplatforming intervention has revealed that user responses to content moderation depend heavily on the specific behavioral dimension being evaluated, with changes in user activity and toxicity proving far more predictable than shifts in community participation diversity. Researchers Benedetta Tessa, Alejandro Moreo, Stefano Cresci, Tiziano Fagni, and Fabrizio Sebastiani formulated the estimation of post-moderation community shifts as an ordinal quantification problem, rather than relying on standard individual-level classification or regression frameworks. By tracking approximately 16 million comments posted over a fourteen-month window spanning seven months before and seven months after the intervention, known colloquially as the Great Ban, the investigators evaluated the predictive informativeness of 753 socio-behavioral, linguistic, relational, and psychological features. Their experimental findings demonstrate that predictive accuracy varies substantially across distinct behavioral outcomes, showing that models equipped with a compact, carefully selected subset of features consistently outperform both single feature categories and models trained on the complete 753-dimensional feature space. This suggests that effective platform moderation strategies cannot rely on uniform assumptions about user reactions, but must instead be calibrated to the specific behavioral targets that governance mechanisms aim to modify.

The investigation was designed to address a fundamental shortcoming in conventional platform governance, where moderation interventions such as content warnings, reach restrictions, temporary suspensions, or outright community bans are routinely deployed in a user-independent, one-size-fits-all fashion without accounting for individual behavioral differences. Drawing on prior empirical observations that different online participants respond disparately to identical enforcement actions—sometimes exhibiting unintended consequences such as platform abandonment, migration toward less regulated alternative spaces, or intensified hostility—the authors sought to identify the specific pre-intervention characteristics most closely associated with subsequent behavioral shifts. The empirical corpus centered on the June 29, 2020 deplatforming action on Reddit, during which fifteen large subreddits were permanently shut down for policy violations, encompassing 2.2 million pre-ban comments posted by active contributors within the targeted communities and 13.8 million comments posted by the same individuals in surrounding, non-banned subreddits across the full observation period. Automated accounts and bots were filtered out through a validated semi-automatic pipeline, and comments made on the exact date of the ban were excluded to prevent transient fluctuations from distorting user behavioral metrics, yielding a final analytical sample of 16,522 users with complete feature profiles.

To formalize the estimation task under realistic operational conditions, the researchers recast the problem within the mathematical framework of quantification, also known as learning to quantify, which focuses on estimating class prevalences across unlabeled populations under dataset shift rather than optimizing individual point predictions. Supervised machine learning algorithms generally depend on the assumption that training and test data are independently and identically distributed, an assumption that routinely breaks down in content moderation environments where deployment distributions diverge systematically from historical training sets. Naive approaches like the classify and count technique, which classify individual instances and sum the resulting category tallies, deliver suboptimal prevalence estimates in the presence of prior probability shift, covariate shift, or concept shift. To establish an evaluation protocol capable of handling ordered outcomes, the authors structured behavioral shifts across activity, toxicity, and participation diversity into five discrete ordinal classes: Highly Decreased, Moderately Decreased, No Variation, Moderately Increased, and Highly Increased. Continuous relative effect scores were discretized using symmetric moderate and high thresholds set respectively at 0.20 and 0.55, which preserved balanced class representation while reflecting practically meaningful behavioral changes, a structure further validated against alternative quantile binning schemes.

Feature engineering encompassed 753 quantitative variables organized into nine predefined, semantically coherent top-level categories, which were further subdivided into 69 discrete subgroups that functioned as the fundamental units of manipulation during heuristic feature optimization. Semantic embeddings constituted the largest single category, comprising 384 dimensions derived by averaging comment-level dense representations extracted via the all-MiniLM-L6-v2 sentence transformer model to capture linguistic and contextual semantics. Psycholinguistic patterns were mapped using 112 features from the LIWC-22 dictionary, covering cognitive processes, affective expressions, and social concerns, while 81 writing style indicators captured part-of-speech distributions and standardized readability indices. User engagement was tracked through 34 activity metrics encompassing comment counts and temporal indicators, alongside 36 relational features measuring interaction structures, influence across banned and non-banned communities, and user initiative. Emotional tone and sentiment were quantified using 32 sentiment features derived from VADER compound polarity scores and emoji lexicons, alongside 19 emotion features from the NRC Emotion Intensity Lexicon, NRC Valence-Arousal-Dominance scales, and EmoAtlas z-scores. Finally, psychological traits were operationalized through 31 socio-psychological metrics capturing inferred demographics, OCEAN personality scores, and Moral Foundations theory dimensions, while 24 toxicity indicators were extracted using the multilingual Detoxify neural classifier across categories including severe toxicity, obscenity, insult, identity attacks, and threats.

Evaluation of baseline quantification algorithms on the complete 753-feature space demonstrated that probabilistic and iterative adjustment methods markedly surpassed simple counting heuristics. Experiments tested Classify and Count (CC), Probabilistic Classify and Count (PACC), and Expectation Maximization for Quantification (EMQ), with the naive Maximum Likelihood Prevalence Estimator (MLPE) serving as a baseline that assumes identically distributed data. Quantification performance was evaluated via the Normalized Match Distance (NMD), a normalized variant of the Earth Mover’s Distance that bounds ordinal error between 0 and 1 while penalizing classification errors proportionally to their distance across the five ordered categories. Using the Artificial Prevalence Protocol (APP), models trained on five independent splits of 8,000 labeled instances across sixteen incremental batch sizes from 500 to 8,000 profiles were challenged against 1,000 synthetic test samples drawn uniformly from the probability simplex to simulate extreme distributional shift. While MLPE and CC yielded high errors that degraded as prior probability shift intensified, EMQ demonstrated consistent performance gains with increasing training sample sizes. Consequently, EMQ paired with an underlying hyperparameter-optimized Logistic Regression classifier was selected as the foundational engine for feature selection and importance estimation.

Because evaluating every combination across 69 feature subgroups represents an intractable combinatorial search space of roughly six hundred quintillion configurations, the researchers implemented a greedy exploration algorithm that prioritized blocks based on their standalone predictive performance. The search began by comparing baseline configurations consisting of each individual top-level group against the complete feature set, revealing that the standalone activity group minimized error for the activity task and the toxicity group minimized error for the toxicity task, whereas the diversity task achieved its best initial performance only when combining all groups simultaneously. Feature subgroups were subsequently evaluated in inverse rank order of their individual loss, allowing candidate blocks to be iteratively added or removed during three initial rounds of exploration, followed by refinement phases restricted purely to feature removal until error reduction plateaued. Models optimized under this greedy regimen, designated as EMQO, yielded substantial performance enhancements over their unpruned counterparts, driving down quantification error by 17.20 percent for activity prediction, 23.76 percent for toxicity prediction, and 18.15 percent for community diversity prediction across the sixteen evaluated training sample sizes.

A critical outcome of the subgroup selection process was the striking disparity in the complexity and concentration of predictive signals across the three behavioral tasks. The toxicity model proved exceptionally parsimonious, requiring only 10 of the 69 available subgroups (14 percent) to achieve its optimal error reduction, whereas the activity model selected 22 subgroups (32 percent) and the participation diversity model incorporated 25 subgroups (36 percent). Analysis of feature importance through the Relative Increase in Error (RIE) showed that predictive power in activity and toxicity was heavily concentrated in specific core predictors: the activity trends subgroup alone accounted for 19.8 percent feature importance in the activity task with a Gini coefficient of 0.65, while the standalone toxicity subgroup contributed 11.4 percent importance in the toxicity task with a Gini coefficient of 0.53. In contrast, participation diversity exhibited a much more diffuse structure characterized by a Gini coefficient of 0.52, where no single subgroup predominated, the overall quantification error remained the highest among all three tasks, and individual feature additions yielded comparatively modest improvements.

Pairwise comparative evaluations confirmed that each behavioral outcome relies on largely distinct, non-overlapping sets of predictive characteristics. Measuring similarity across selected feature configurations yielded an average Jaccard index of 0.22 and a mean Rank-Biased Overlap (RBO) of 0.19, with activity and diversity showing the closest correspondence while activity and toxicity exhibited the lowest similarity. Task-specific allocations demonstrated notable theoretical alignments: relational features capturing community interactions were entirely absent from the activity and toxicity models but fully incorporated within the diversity task, where structural engagement directly defines cross-community movement. Conversely, sentiment features were selected predominantly for activity estimation, aligning with psychological reactance principles wherein heightened negative pre-ban affect correlates with vigorous protest activity or complete withdrawal. Writing style subgroups were selected at a 60 percent rate for diversity compared to 25 percent for activity and 20 percent for toxicity, reflecting the distinct linguistic norms across subreddits, while emotional intensity features were selected exclusively for the diversity task, matching the Broaden-and-Build hypothesis that positive affective states facilitate broader cross-community exploration while negative states induce defensive contraction.

Despite this marked task dependence, the empirical analysis revealed a compact core of staple feature subgroups that proved consistently informative across all three prediction tasks: mentions and links, temporal trends, punctuation patterns, and baseline toxicity. The universal selection of mentions and external links underscores the predictive importance of social embeddedness and external referential behavior, while longitudinal trend indicators confirm that pre-intervention behavioral trajectories strongly signal post-moderation adaptation. The cross-task utility of punctuation features reinforces computational sociolinguistic evidence that non-alphanumeric markers serve as pragmatic signals of stance, tone, and identity. Meanwhile, pre-existing toxicity levels remained predictive of subsequent activity volumes and participation breadth in addition to future toxic language. Conversely, 33 percent of all evaluated feature subgroups were never selected in any task configuration, demonstrating that many popular features frequently utilized in online hate speech detection and behavioral modeling offer negligible explanatory utility for post-moderation adaptation forecasting.

To interpret why certain features were excluded and evaluate whether alternative feature combinations could achieve equivalent predictive utility, the authors performed a task-independent redundancy analysis using Canonical Correlation Analysis (CCA) standardized across the first five principal components of each block, complemented by two-dimensional t-SNE projections of the correlation matrix. This analysis confirmed pervasive, structured redundancy spanning both within and across nominal feature groups, exemplified by dense clusters of near-interchangeable sentiment and emoji metrics where incorporating one block rendered neighboring blocks redundant during greedy search selection. The authors noted several key limitations to their findings, emphasizing that the empirical evidence reflects observational correlations rather than causal mechanisms, lacks longitudinal time-series modeling, and derives from a single platform undergoing a specific deplatforming event in 2020. Nevertheless, the study provides robust empirical evidence that post-moderation user dynamics cannot be adequately managed through static, one-size-fits-all policies or monolithic machine learning predictors, highlighting the necessity of designing objective-specific platform governance systems that align tailored interventions with the precise behavioral outcomes platforms seek to guide.

Subject of Research: Quantification of feature importance for predicting post-moderation online user behavioral changes

Article Title: Quantifying feature importance for online content moderation

Article References: Tessa, B., Moreo, A., Cresci, S., Fagni, T., & Sebastiani, F. (2026). Quantifying feature importance for online content moderation. EPJ Data Science, 15(1), Article 70. https://doi.org/10.1140/epjds/s13688-026-00659-w

Image Credits: AI Generated

DOI: 10.1140/epjds/s13688-026-00659-w

Keywords: content moderation, ordinal quantification, social media analytics, feature selection, behavioral prediction, Reddit Great Ban, toxicity estimation, algorithmic governance, Quantifying, feature, importance, online