Researchers have developed a practical framework for measuring how well clinical prediction models distinguish between patients who will experience many repeated medical events and those who will experience few, addressing a long-standing gap in how recurrent outcomes such as seizures and asthma exacerbations are evaluated. The work, published in Diagnostic and Prognostic Research, provides adapted discrimination statistics together with open-source R code that any research team can apply to their own recurrent event models.
Clinical prediction models combine multiple pieces of patient information to forecast a clinical outcome, and their performance is conventionally judged on two fronts: calibration, which asks whether predicted probabilities agree with observed ones, and discrimination, which asks how well the model separates individuals who experience the outcome from those who do not. Discrimination is a core metric recommended by the TRIPOD+AI reporting guidelines, yet while methods for evaluating calibration of recurrent event models have been developed, approaches for assessing discrimination in this setting have remained limited. Recent contributions, including concordance-based methods and Brier-type accuracy measures, address parts of the problem, but no widely adopted, flexible framework for discrimination has existed until now.
The observed outcome for each individual is defined as the total number of events experienced during a specified follow-up period, for example the number of asthma exacerbations within one year of study entry, while the prediction is the expected cumulative number of events over the same horizon. For every possible pair of patients, the pair is counted as concordant if the person with the higher predicted event count also had the higher observed count, discordant if the ordering is reversed, and tied if either the predicted or observed counts are equal.
How ties are handled turns out to matter considerably, because tied values occur substantially more often when event counts rather than binary outcomes are compared. A tie provides no evidence that the model correctly ranked risk between two individuals, but different statistics treat it differently. Kendall’s tau-a, the most conservative option, counts ties as discordant and shrinks the measure toward zero. Goodman-Kruskal’s gamma ignores ties entirely, which can make it appear artificially high when ties are frequent. Somers’ D takes a middle path, treating outcome ties as incomparable and scoring prediction ties as half a concordant pair, and it converts easily into the familiar concordance statistic, where a value of 0.5 corresponds to flipping a coin. The researchers implemented all four tie-handling variants so that analysts can choose the metric suited to their question.
Quantifying the uncertainty of these estimates posed its own challenge. Confidence intervals for discrimination are usually obtained by bootstrap resampling, but the abundance of ties in recurrent event data reduces the number of informative pairwise comparisons and can lead bootstrap procedures to underestimate variability. The team therefore recommends jackknife resampling, in which the dataset is repeatedly re-sampled by omitting observations, treating the patient rather than the individual event as the unit of deletion to respect the correlation structure of recurrent events. Because dropping a single patient at a time can yield unrealistically narrow intervals, they additionally employed a delete-d grouped jackknife, in which larger subsets of subjects, here 10 percent of the cohort across 200 replicates, are omitted at each iteration. Intervals can then be constructed using normal approximations based on pseudo-values, the median absolute deviation, or the interquartile range of the replicate estimates.
To demonstrate the approach, the researchers applied four recurrent event models to two clinical datasets with very different event rates. The models compared were the negative binomial and zero-inflated negative binomial count models, the Andersen-Gill extension of the Cox model, and the Prentice-Williams-Peterson Total Time model, which stratifies by event order and allows the baseline hazard to vary between successive events. The asthma data came from the Optimum Patient Care Research Database, a large UK primary care database, including patients aged 12 to 80 with diagnosed asthma, with exacerbations defined according to European Respiratory Society and American Thoracic Society guidelines as hospital admissions, emergency attendances, or acute consultations resulting in an oral corticosteroid prescription. The epilepsy data came from the SANAD trials, in which 702 participants with generalised or unclassifiable epilepsy were randomised between valproate, lamotrigine and topiramate, with seizure recurrence examined over two years following randomisation.
The results revealed striking differences between models. In the asthma data, the Prentice-Williams-Peterson model showed the strongest discrimination, with a concordance statistic of 0.939 and a grouped-jackknife 95 percent confidence interval of 0.935 to 0.944, while the rank-based metrics, Kendall’s tau-a of 0.603 and Somers’ D and gamma both at 0.879, also indicated very strong discrimination. The Andersen-Gill model performed moderately well with a concordance statistic of 0.823, whereas the negative binomial model at 0.573 and the zero-inflated negative binomial model at 0.586 showed weak discrimination, barely better than chance. In the epilepsy data the same pattern emerged: the Prentice-Williams-Peterson model again performed best, with a concordance statistic of 0.943 and all rank-based metrics above 0.85, while the negative binomial, zero-inflated negative binomial and Andersen-Gill models showed moderate discrimination, with concordance statistics around 0.64 to 0.65.
The resampling comparison proved just as instructive as the model comparison. Standard leave-one-out jackknife intervals were frequently extremely narrow or degenerate, collapsing to the point estimates, particularly in the large asthma dataset, whereas the grouped delete-d jackknife produced wider and more informative intervals, especially for models with weaker discrimination or more heterogeneous predicted risk. In the epilepsy data, for example, the negative binomial model’s concordance estimate of 0.654 carried a grouped-jackknife interval stretching from 0.537 to 0.771, a far more honest reflection of sampling variability. The authors caution, however, that the coverage properties of these intervals were not evaluated, so it remains unclear which approach comes closest to the nominal 95 percent level, and they call for simulation studies to investigate coverage and width under varying numbers of ties, event frequencies and sample sizes.
The team is careful about what the findings do and do not imply. Although the Prentice-Williams-Peterson model ranked individuals most accurately in both examples, the authors stress that this should not be read as a general recommendation to prefer that model in all settings; model choice should be guided by the scientific question, the underlying event process and model fit. The analyses also evaluated apparent discrimination within the same datasets used for model development, so the estimates may be optimistic, and further work on internal and external validation remains important. The two illustrative datasets may not generalise to other diseases or settings, since relative model performance can vary with event frequency, follow-up duration and within-patient dependence.
What the study delivers is a set of practical tools. The discrimination measures assess a fixed prediction horizon, comparing cumulative event burden within a specified window, and the underlying computations were engineered for efficiency, with a C++ implementation that evaluated concordance for 100,000 patients in roughly 29 seconds. The researchers suggest that analysts select a small number of appropriate metrics aligned with their research question and data structure rather than applying all measures at once, and they encourage use of the methods in both model development and external validation. Used alongside existing calibration tools, the framework supports comprehensive performance evaluation and could help standardise the validation and reporting of prediction models built for recurrent clinical events, from repeated seizures to repeated exacerbations, in line with TRIPOD+AI recommendations.
Subject of Research: Statistical methods for evaluating the discrimination of clinical prediction models for recurrent events
Article Title: Evaluating discrimination of prediction models for recurrent clinical events
Article References: Spain, T. J., Hunt, A., Sudell, M., Hutton, J. L., Putter, H., Watson, V., Blakey, J., Marson, A., & Bonnett, L. J. (2026). Evaluating discrimination of prediction models for recurrent clinical events. Diagnostic and Prognostic Research, 10(1), Article 23. https://doi.org/10.1186/s41512-026-00238-7
Image Credits: AI Generated
DOI: 10.1186/s41512-026-00238-7
Keywords: prediction models, recurrent events, discrimination, concordance statistic, jackknife resampling, epilepsy, asthma, TRIPOD+AI, Andersen-Gill, Prentice-Williams-Peterson, clinical prediction, biostatistics
News Source: Ophelia Keating. (October 28, 2026). New statistical tools measure how well models predict repeated medical events. Scienmag.
