Prognostic COPD Model Validation Reveals Calibration Drift Across Diverse Clinical Settings

A comprehensive validation study evaluating prognostic models for chronic obstructive pulmonary disease has demonstrated that while predictive discrimination remains remarkably consistent across different time periods and geographical regions, calibration degrades substantially due to underlying shifts in baseline mortality risk. Published in Diagnostic and Prognostic Research, the investigation led by Haya Elayan and colleagues replicated an established logistic regression model designed to predict five-year mortality from the date of diagnosis across tens of thousands of patients in England. Across diverse external validation cohorts, the reproduced candidate model achieved area under the receiver operating characteristic curve values between 0.79 and 0.80, confirming stable discriminative ability. However, calibration-in-the-large varied from 0.001 to 0.455, indicating that temporal changes in healthcare practices and regional differences introduce significant prediction drift. To address this performance gap, researchers evaluated methods to incorporate ancillary data while accounting for heterogeneity.

Chronic obstructive pulmonary disease represents a progressive lung condition characterized by persistent airflow limitation and chronic airway inflammation. Within the United Kingdom, epidemiological estimates indicate that the condition affects approximately one to two percent of the population. Clinical symptoms develop gradually and worsen over time, commonly presenting as chronic cough, shortness of breath, wheezing, chest tightness, frequent respiratory infections, fatigue, and unintended weight loss. As lung damage advances, routine daily activities become increasingly difficult, impairing patient quality of life. Diagnosis involves medical history evaluation, physical examination, imaging, and lung function tests like spirometry. Management focuses on controlling symptoms, preventing exacerbations, and slowing progression through bronchodilators, steroids, pulmonary rehabilitation, oxygen therapy, and lifestyle changes like smoking cessation. Prognostic models estimate individual risk of future outcomes based on clinical information, assisting healthcare practitioners with treatment planning, care administration, and disease management.

Prognostic models predicting long-term outcomes face challenges related to temporal drift and regional variations. Models predicting five-year mortality require extended follow-up, necessitating development cohorts diagnosed ten to twenty years prior to publication. Improvements in medical treatments, evolving healthcare delivery, and changes in coding practices can cause older models to produce miscalibrated risk predictions that misestimate mortality in contemporary or geographically distinct target populations. Developing a new model solely using data from the target population poses severe difficulties when target datasets have small sample sizes, which heightens the risk of statistical overfitting. Utilizing related ancillary data from another population, time period, or healthcare setting offers a viable solution, but differences in underlying distributions can induce data distribution shifts. These discrepancies arise from case-mix shifts in predictor distributions, event rate shifts in outcome prevalence, or changes in predictor-outcome associations, all of which can degrade predictive performance.

To examine these challenges, the researchers extracted records from the Clinical Practice Research Datalink GOLD database linked to Office for National Statistics mortality data and Hospital Episode Statistics. The study reproduced a prognostic model previously developed by Kiddle and colleagues, which uses logistic regression to predict binary five-year mortality from the date of chronic obstructive pulmonary disease diagnosis. Ascertainment of five-year vital status across linked practices enabled complete binary outcome definitions. The reproduced cohort comprised 55,593 patients diagnosed between January 1, 2004, and September 19, 2012, across English practice regions. Model covariates included patient age, sex, body mass index, forced expiratory volume in one second percentage predicted, and smoking status. Age, body mass index, and lung function were mean-centered with squared terms added. Because missingness was substantial for body mass index and lung function, median imputation and missingness indicators were implemented, while missing smoking status values were imputed via random sampling.

The investigation established four study settings to evaluate model behavior under temporal and geographic comparisons. In the Temporal-New setting, the candidate model fitted on the 55,593 patients from 2004 to 2012 served as ancillary data, while 10,907 patients diagnosed between September 20, 2012, and December 30, 2014, formed the prospective target dataset. The NorthEast-New setting evaluated regional variation by fitting a model on 53,728 patients outside the North East and testing on 235 contemporary North East patients to assess small sample dynamics. The London-New setting fitted models on 50,985 patients outside London and validated against 1,492 contemporary London patients. Lastly, the Temporal-Existing setting evaluated whether dividing the original development data temporally could address drift from the outset. In that setting, 39,068 patients diagnosed between 2004 and 2009 served as ancillary data, while 16,525 patients diagnosed between 2010 and 2012 served as target data, with subsequent external validation on the 10,907 patients diagnosed from 2012 to 2014.

The researchers tested several statistical approaches to incorporate ancillary data while accounting for heterogeneity across the target settings. Target-only logistic regression fitted an unweighted model solely using target data. Intercept recalibration retained original model coefficients and updated only the intercept using the target data, with the linear predictor entered as an offset variable constrained to a coefficient of one, implemented using all development data or ancillary data alone. Logistic recalibration updated both the intercept and overall calibration slope by fitting a logistic regression on target data using the linear predictor as the sole variable. Membership-based recalibration assigned weights to ancillary individuals using propensity scores reflecting the conditional probability of belonging to the ancillary dataset based on observed predictors. These propensity weights were adjusted by a sample size ratio forgetting factor to balance cohort sizes, followed by adding a target membership dummy variable to adjust calibration-in-the-large.

External validation of the candidate model demonstrated stable discrimination but significant calibration shifts across target settings. In Temporal-New validation, the candidate model attained an area under the curve of 0.793, with a 95 percent confidence interval of 0.782 to 0.802. However, mean calibration-in-the-large was 0.283, with a confidence interval of 0.228 to 0.343, reflecting a 3.0 percentage point difference between expected and observed survival, indicating risk overestimation. In the London-New setting, discrimination was 0.803, but calibration-in-the-large reached 0.455, producing a 5.8 percentage point miscalibration. Conversely, in the NorthEast-New setting, the candidate model achieved an area under the curve of 0.786 and a calibration-in-the-large of 0.001, corresponding to a negligible 0.04 percentage point survival difference. These findings demonstrate that while discriminatory ranking of patient risks remained reliable across regions and periods, changes in baseline event rates created notable miscalibration in prospective and urban cohorts.

Evaluating alternative modeling methods in newly available target datasets showed that recalibration strategies successfully resolved baseline calibration errors. In the Temporal-New dataset, all alternative models achieved optimism-adjusted calibration-in-the-large values of minus 0.003, improving upon the candidate model’s 0.283, while maintaining discrimination near 0.79 and yielding slightly lower Brier scores than the candidate baseline of 0.134. In the London-New setting, all alternative methods shifted calibration-in-the-large toward zero compared to the candidate model’s 0.455, while improving Brier scores relative to the baseline of 0.128. Logistic recalibration models achieved calibration slopes closest to 1 in London, while intercept recalibration variants produced narrower bootstrap confidence intervals for test calibration slopes across 200 iterations. Applying intercept recalibration did not harm performance metrics even in the North East setting where baseline recalibration was not strictly necessary, demonstrating the broad utility of updating intercept terms when adjusting models to local target conditions.

The comparative analysis revealed notable performance limitations for target-only estimation and propensity-based weighting when applied to small sample sizes. In the NorthEast-New cohort of 235 patients, developing a model solely on target data caused discrimination to drop to an area under the curve of 0.727, accompanied by an unstable calibration slope of 0.29. The membership-based recalibration model similarly showed lower discrimination at 0.753 and an attenuated calibration slope of 0.772, alongside wider bootstrap confidence intervals. These lower metrics reflected sample size instability and the down-weighting of ancillary records. In contrast, intercept and logistic recalibration models preserved higher discrimination than the candidate model in the North East, with calibration-in-the-large near minus 0.02. Intercept recalibration models also maintained narrower bootstrap test intervals for calibration slope, showing that constraining predictor coefficients from large ancillary pools stabilizes predictions when target cohorts are small.

In the Temporal-Existing setting, the researchers examined whether applying recalibration methods across a temporal split of the historical development data could mitigate calibration drift from the outset. Internal optimism-adjusted validation on the 2010 to 2012 target data demonstrated similar performance across all models, with discrimination around 0.796 to 0.797, Brier scores near 0.133 to 0.134, and calibration-in-the-large near zero. External validation on subsequent patients diagnosed between 2012 and 2014 demonstrated that all alternative methods improved calibration compared to the candidate model. The candidate model exhibited a calibration-in-the-large of 0.283 and overestimated mortality by 3.0 percentage points, whereas models updated during development achieved calibration-in-the-large values between 0.066 and 0.069, underestimating survival by only 0.8 to 0.9 percentage points. These findings suggest that incorporating temporal structures within available development data reduces the impact of temporal heterogeneity, although the authors cautioned that observed differences could reflect random variation or specific dataset characteristics.

The study authors discussed these findings in relation to broader research on model adaptation, noting regional recalibration efforts in SCORE2, model updating by Kappen and Janssen, and evaluations of Framingham, BODE, and PRO-MEL tools across diverse populations. The authors also highlighted several study limitations. The reproduced candidate model was published recently and is not yet implemented in clinical practice, which may moderate the immediate impact compared to updating established tools. Data extraction constraints yielded higher missingness for lung function and body mass index than in the original study, necessitating median imputation rather than multiple imputation. Additionally, the study evaluated existing models and recalibration methods rather than specifically investigating the underlying drivers of miscalibration. Nevertheless, the investigation demonstrates that intercept recalibration offers a simple, effective method to restore baseline calibration across settings, while membership-based weighting provides a viable alternative, equipping researchers with practical strategies to manage population heterogeneity.

Subject of Research: Prognostic model validation and recalibration methods in chronic obstructive pulmonary disease

Article Title: Validation of a prognostic model for chronic obstructive pulmonary disease (COPD) and evaluation of methods to include related data

Article References: Elayan, H., Broomfield, J., Rutherford, M. J., Sperrin, M., Martin, G. P., Jenkins, D. A., & Booth, S. (2026). Validation of a prognostic model for chronic obstructive pulmonary disease (COPD) and evaluation of methods to include related data. Diagnostic and Prognostic Research, 10(1), Article 31. https://doi.org/10.1186/s41512-026-00245-8

Image Credits: AI Generated

DOI: 10.1186/s41512-026-00245-8

Keywords: chronic obstructive pulmonary disease, prognostic modeling, model validation, intercept recalibration, calibration drift, CPRD GOLD, mortality prediction, propensity score weighting, Validation, prognostic, model, chronic