Scaled Brier Score Outperforms Common Metrics When Prevalence Shifts in Classification Tests

When a machine learning model or diagnostic test labels every case correctly, most performance metrics will celebrate. But what happens when the underlying rate of the condition being predicted — the prevalence — changes? A new simulation study published in Discover Data on 17 September 2026 argues that the answer matters far more than many practitioners realize, and that one metric in particular, the scaled Brier score, behaves more reliably than the widely promoted alternatives.

The metrics examined included accuracy, balanced accuracy, the Brier and scaled Brier scores, the F score, Cohen’s Kappa and prevalence-adjusted and bias-adjusted Kappa (PABAK), the Matthews correlation coefficient (MCC), receiver operating characteristic and precision-recall area under the curves (ROC-AUC and PR-AUC), and the Youden Index. The motivation was straightforward: several recent studies have recommended the MCC as the preferred measure of hard classification accuracy, yet the authors suspected that its mathematical origins might limit its interpretability for classification tasks.

The MCC is derived from the Pearson correlation between two binary categorical variables, which gives it fixed symmetric bounds of minus one and plus one, with a value of zero indicating a lack of relationship. The authors point out that this zero point is problematic for classification. In a dataset where a model predicts zero for every observation, the MCC reports a value of zero, which is conventionally read as indicating no relationship. The scaled Brier score, by contrast, is probability-based: a value of zero indicates a random association, meaning predictions are no better than simply guessing the sample prevalence for each observation, and negative values indicate predictions that are worse than random. In that same all-zeros scenario, the scaled Brier score correctly reports a negative value, because no genuine attempt at prediction was made.

The scaled Brier score itself is a generalization of the Brier score, which was originally created to evaluate continuous predicted probabilities against binary outcomes as a mean squared error. For hard classifications, zero and one values can be treated as zero percent and one hundred percent predicted probabilities. The raw Brier score can then be scaled to the sample’s prevalence by dividing by Briermax, the maximum obtainable from a non-informative, intercept-only model, which equals prevalence multiplied by one minus prevalence. The result is a metric with an upper bound fixed at one, a lower bound that varies with prevalence, and a zero point anchored to randomness rather than to the absence of a correlation.

To compare metrics systematically, the researchers proposed three prevalence-related criteria. First, metric values should not be identical when prevalence changes. Second, as prevalence increases, metric values should decrease for scenarios where every observation is labelled a non-case, because it becomes easier to correctly identify cases when they are common. Third, as prevalence increases, metric values should increase for scenarios where every observation is labelled a case, because it becomes harder to correctly identify non-cases when cases dominate. Poor accuracy was defined as values below zero for Kappa, PABAK, Youden Index, MCC, and the scaled Brier score, and below 0.50 for balanced accuracy and ROC-AUC, with accuracy, one minus Brier, F score, and PR-AUC also judged against a 0.50 threshold to reflect common practice.

The binary simulations used 50,000 observations per scenario, with case prevalence varying at 10, 50, and 90 percent, sensitivity varying from zero to one hundred percent, and specificity or positive predictive value also varied across two series of scenarios. The results were striking. Only the scaled Brier score satisfied all three proposed criteria. Accuracy, one minus the Brier score, and PABAK satisfied the first criterion and partially satisfied the second and third, but their generated values sometimes led to incorrect conclusions. For example, when sensitivity was zero percent and specificity was one hundred percent — meaning every case was missed — accuracy and one minus Brier both reported 0.90 at a prevalence of ten percent, and PABAK reported 0.80, falsely suggesting very high accuracy. Only the scaled Brier score reported values below random at every prevalence and moved in the correct direction, with values of minus 0.11, minus 1.00, and minus 9.00 as prevalence rose from ten to ninety percent.

The pattern repeated in the opposite direction. When sensitivity was one hundred percent and specificity was zero percent — every observation labelled a case — balanced accuracy, ROC-AUC, Youden Index, Kappa, and MCC all returned unchanging values as prevalence increased, while accuracy, one minus Brier, the F score, PR-AUC, and PABAK increased but still reported high values that mischaracterized the predictions. The scaled Brier score alone reported below-random values at every prevalence, rising from minus 9.00 at ten percent prevalence to minus 0.11 at ninety percent, correctly signalling extremely poor accuracy in both directions. The authors note that a value of minus 9.00 indicates that not a single case, or not a single non-case, was correctly predicted depending on the scenario.

The multiclass simulations extended the analysis to three-category data with 50,000 observations, using the Rk formulation of MCC and micro, macro, and weighted variants of several metrics. Because modifying sensitivity, specificity, or positive predictive value for one class alters the counts in the others, the exact binary criteria could not be applied directly, so the researchers designed approximations. In the second-criterion approximation, where classes became increasingly unbalanced and only a class not among the true labels was predicted, only the scaled Brier score decreased in value and indicated poor accuracy, falling from minus 2.00 to minus 9.81, while accuracy, one minus Brier, balanced accuracy, Kappa, and MCC all reported zero and PR-AUC weighted rose from 0.67 to 0.91. In the third-criterion approximation, where all predictions were assigned to one class of rising prevalence, only the scaled Brier score increased while indicating poor accuracy, moving from minus 2.08 to minus 0.08, whereas several other metrics reported high values at ninety percent prevalence.

The practical implications are considerable. The authors illustrate with a diagnostic test example: if sensitivity and specificity are both 95 percent, balanced accuracy and ROC-AUC report 0.95 and the Youden Index 0.90 regardless of whether prevalence is 50 percent or one percent, yet the positive predictive value swings from 95 percent to 16 percent. The scaled Brier score, by contrast, is 0.80 at 50 percent prevalence and minus 4.05 at one percent, because it accounts for prevalence directly. The authors caution that by ignoring prevalence, healthcare decision-makers may incorrectly consider a diagnostic test acceptable in a population where the condition is rare. They also note that PR-AUC lacks a fixed baseline indicating randomness, since its baseline is the sample prevalence, and that both PR-AUC and the F score exclude true negatives from their calculations.

The study also acknowledges limitations. The scaled Brier score’s calculated value is not directly comparable across different levels of prevalence, though the authors note that if sensitivity and specificity are known, data can be simulated and the score recalculated at another prevalence level. Multiclass scaled Brier scores are global values that may mask poor performance in individual categories, as demonstrated by an example where a global score of 0.658 concealed a below-random score of minus 0.11 for one class, and the authors recommend supplementing multiclass scores with separate binary scores per class. Like most metrics, misclassifications are equally weighted, and the multiclass criteria were only approximations of the binary criteria. Nevertheless, the authors conclude that the recommendation of the scaled Brier score as a measure of predictive accuracy for continuous probabilities can be extended to hard classification data, and that for most scenarios simulated, other metrics reported substantially higher values that could greatly change the conclusions drawn.

Subject of Research: Impact of prevalence on classification performance metrics, comparing the scaled Brier score with the Matthews correlation coefficient and other measures

Article Title: Evaluating the impact of prevalence on the scaled Brier score, Matthews correlation coefficient, and other performance metrics for binary and multiclass hard classifications

Article References: Lambert, P. J., Pitz, M., Singh, H., & Decker, K. M. (2026). Evaluating the impact of prevalence on the scaled Brier score, Matthews correlation coefficient, and other performance metrics for binary and multiclass hard classifications. Discover Data, 4(1), Article 21. https://doi.org/10.1007/s44248-026-00119-w

Image Credits: AI Generated

DOI: 10.1007/s44248-026-00119-w

Keywords: scaled Brier score, Matthews correlation coefficient, classification metrics, prevalence, binary classification, multiclass classification, ROC-AUC, PABAK, simulation study, predictive accuracy, confusion matrix, hard classification