Predictive Model for Nursing Home Dependency Loses Accuracy Outside VA Data

A machine learning tool designed to predict future dependency in daily activities among nursing home residents has been put to one of the toughest tests in clinical artificial intelligence: external validation in a completely different population. The Computational Barthel Index Tool, or CBIT, was originally built from data on U.S. Department of Veterans Affairs patients and can forecast declines in activities of daily living up to one year ahead. In a new study, researchers examined how well the pretrained models performed on a large, non-VA nursing home cohort, and whether adapting them to the new population could restore their accuracy.

The validation relied on SEER-Medicare linked data covering 843,836 patients and more than 8 million Minimum Data Set assessments. The team reconstructed the predictor and outcome structures of the original CBIT models within this cohort and then evaluated the unmodified VA-trained models for discrimination and calibration at four prediction horizons: current status, 3 months, 6 months, and 12 months.

Under strict external validation, discrimination dropped substantially compared with the original VA models, and the decline was most pronounced for longer-term predictions. Mean AUC for the evaluation models fell from 0.795 in the original models to 0.693 at current prediction, 0.595 at 3 months, 0.595 at 6 months, and 0.592 at 12 months. Re-evaluation models retained stronger performance but showed a similar downward pattern across timeframes.

Calibration also deteriorated, with calibration error rising particularly for evaluation models at the longer prediction horizons. Subgroup analyses revealed only modest variation in discrimination across demographic and clinical groups, indicating that the performance loss reflected a broad transportability problem rather than failure in specific patient subpopulations.

The most encouraging finding came from population-specific adaptation. When the researchers progressively incorporated larger subsets of SEER-Medicare training data into the pretrained models, discrimination recovered substantially, improving as more local data were used. With the full adaptation training set, the adapted models achieved performance comparable to models reconstructed entirely from SEER-Medicare data.

The authors conclude that transporting clinical machine learning models across populations can carry a substantial cost in accuracy, and they propose a structured workflow: externally validate pretrained models first, then adapt them when performance degradation is observed. Full model reconstruction remains a viable alternative where sufficient local data are available. For nursing home care planning, the results suggest that VA-derived prediction tools should not be deployed unchanged in other populations without local validation and adaptation.

Subject of Research: External validation and adaptation of a machine learning tool predicting activities of daily living dependency in nursing home populations

Article Title: External validation and adaptation of the Computational Barthel Index Tool in a large nursing home cohort

Article References: Ngana, L. P., Wojtusiak, J., Haghighathoseini, A., Patel, B., & Desai, D. (2026). External validation and adaptation of the Computational Barthel Index Tool in a large nursing home cohort. PLOS Digital Health, 5(10), e0001793. https://doi.org/10.1371/journal.pdig.0001793

Image Credits: AI Generated

DOI: 10.1371/journal.pdig.0001793

Keywords: machine learning, nursing homes, Barthel Index, external validation, SEER-Medicare, Veterans Affairs, activities of daily living, model transportability, prediction models, Minimum Data Set, calibration, clinical AI