Hourly vasopressor prediction model shows setting-dependent calibration across three databases

A retrospective study evaluating an hourly model for predicting documented continuous vasopressor initiation across three databases found that discrimination transferred unevenly, while calibration and alert performance remained setting-dependent. The research, published in Frontiers in Medicine, utilized data from MIMIC-IV, eICU-CRD, and SICdb to assess the transportability of clinical prediction models. The authors concluded that the evaluated model and its associated alert policies are not ready for clinical deployment, emphasizing that offline assessment against prespecified clinical criteria must precede any prospective silent evaluation in live clinical settings.

Clinical prediction models may retain discrimination across settings while losing probability calibration or a workable alert policy. To address this, the researchers evaluated an hourly prediction model designed to forecast the first documented initiation of continuous vasopressor therapy. The outcome was defined as the first initiation within six hours of the prediction landmark and before the patient’s exit from the ICU or unit. If a patient exited the unit without vasopressor initiation, this was treated as an operational non-event for the purposes of the analysis.

The data sources included MIMIC-IV version 3.1, eICU-CRD version 2.0, and SICdb version 1.0.8. The study included untreated adult stays that contributed hourly landmarks from ICU hour 6 through hour 24 while the patients were still under observation. Histogram-based gradient boosting (HGB) and logistic regression models were fitted and selected using patient-disjoint groups from the MIMIC-IV reference period. Evaluation was then conducted on a separate MIMIC-IV group and the two external databases to test the model’s transportability. Source observations were range-checked before predictor summarization, and SICdb hourly aggregates were withheld until the source hour had ended.

The evaluation encompassed a substantial number of data points, including 165,204 landmarks from 9,534 MIMIC-IV reference-period test stays, 2,745,224 landmarks from 168,636 eICU-CRD stays, and 58,756 landmarks from 3,997 SICdb stays. The HGB model achieved Area Under the Receiver Operating Characteristic Curve (AUROC) values of 0.770, 0.849, and 0.695 in the MIMIC-IV, eICU-CRD, and SICdb datasets, respectively.

Calibration performance, however, showed significant variability. Across 100 identifier-disjoint local-update splits, the median Brier skill relative to a calibration-subset constant forecast was -0.168, -0.005, and 0.023 before updating. After full logistic recalibration, these values shifted to 0.019, 0.025, and 0.028. This suggests that while recalibration improved the model’s performance in some settings, the calibration remained setting-dependent and did not uniformly transfer.

The study also examined the alert burden associated with different threshold policies. With thresholds targeting five raw alerts per 100 calibration rows, event detection after six-hour suppression was 0.406, 0.510, and 0.275 in the three datasets. The number of false emitted alerts was 1.64, 1.52, and 1.51 per 100 evaluation rows, respectively. When thresholds were adjusted to target 80% raw landmark sensitivity, the model detected 0.885, 0.873, and 0.883 of eligible events after suppression. However, this higher sensitivity came at the cost of a significant increase in false alerts, with 63.8%, 50.7%, and 86.9% of event-free stays receiving at least one false alert.

The authors noted that these retrospective timing rules did not verify the availability of data in live clinical information systems. The authors emphasize that the model’s inability to maintain calibration and a workable alert policy across settings is a significant barrier to its clinical utility.

The findings highlight the challenges of transporting clinical prediction models across different healthcare environments. The study underscores the need for careful offline assessment and validation against prespecified clinical criteria before any prospective evaluation.

In conclusion, the study provides a detailed evaluation of the transportability of an hourly vasopressor initiation prediction model. The results indicate that while the model can be adapted to different datasets, its performance is highly dependent on the specific setting. The lack of consistent calibration and the high burden of false alerts in some settings suggest that the model is not currently suitable for clinical deployment.

Subject of Research: Transportability, calibration, and alert burden of an hourly model for documented continuous vasopressor initiation: a retrospective three-database study

Article Title: Transportability, calibration, and alert burden of an hourly model for documented continuous vasopressor initiation: a retrospective three-database study

Article References: Xu, X., Chen, T., Shu, P., Liu, Z., Ying, F., Bao, C., Chen, C., & Sun, J. (2026). Transportability, calibration, and alert burden of an hourly model for documented continuous vasopressor initiation: a retrospective three-database study. Frontiers in Medicine, 13, Article 1993720. https://doi.org/10.3389/fmed.2026.1993720

Image Credits: AI Generated

DOI: 10.3389/fmed.2026.1993720

Keywords: Transportability, calibration, alert, burden, hourly, model, documented, continuous, vasopressor, initiation, retrospective, three-database