Pooling Trial And Observational Data Improves Counterfactual Machine Learning Model Evaluation

When machine learning models are deployed within complex healthcare environments to guide clinical interventions, practitioners frequently seek to answer counterfactual questions concerning patient outcomes under hypothetical treatment strategies. Evaluating such algorithmic predictions becomes fraught with statistical risk whenever the data used to train or evaluate a system differs from the target clinical population in treatment assignment patterns or baseline patient characteristics. In a methodological study published in Diagnostic and Prognostic Research, a research team led by Sarah C. Voter, Issa J. Dahabreh, Christopher B. Boyer, Habib Rahbar, Despina Kontos, and Jon A. Steingrimsson developed an analytical framework for counterfactual prediction that systematically integrates randomized trial data with observational registries. Their research rigorously compares standalone observational analyses with transportability approaches that project trial findings onto target populations, while introducing a joint estimation framework that pools both data sources. By delineating the precise identifiability conditions required for valid counterfactual risk estimation, the investigators demonstrate how combining diverse clinical data streams can improve statistical efficiency without sacrificing inferential validity.

The fundamental challenge addressed by the authors stems from the distinct trade-offs inherent in clinical trials versus real-world health records when training predictive algorithms. While factual predictions depend entirely on observed variables and measurable historical events, counterfactual predictions address hypothetical what-if scenarios regarding how patients would fare under specific, uniformly enforced clinical decisions. Machine learning algorithms designed to predict risk under possible treatment regimes must contend with datasets that reflect disparate selection mechanisms. Randomized controlled trials provide robust protection against treatment confounding because therapeutic allocation is determined by design, ensuring that treated and untreated individuals remain conditionally exchangeable within baseline covariate strata. However, clinical trials rarely sample randomly from general populations; instead, they enroll restrictive convenience samples governed by strict eligibility rules, which often leads to the documented under-representation of marginalized communities and complex comorbidity profiles. Conversely, observational databases such as electronic health records or medical claims encompass diverse real-world populations, yet therapy decisions within routine care reflect clinician judgment and unmeasured health indicators that violate basic exchangeability assumptions.

To resolve these conflicting evidentiary limitations, the authors formalize the mathematical requirements for fitting and evaluating a tailored machine learning model designed to estimate the conditional expectation of potential outcomes in an observational target population. Under an observational analysis strategy, investigators rely exclusively on real-world registry data to infer counterfactual performance, which requires satisfying three critical identifiability criteria labeled assumptions A1, A2, and A3. Assumption A1 enforces consistency, ensuring that an individual’s observed health outcome matches their potential outcome under the assigned intervention, thereby ruling out hidden variations in treatment delivery, patient interference, or observational surveillance artifacts. Assumption A2 mandates conditional exchangeability between treatment groups given baseline covariates, asserting that no unmeasured confounders simultaneously govern therapy selection and clinical outcomes. Assumption A3 requires positivity of treatment assignment, ensuring that every patient subgroup defined by baseline covariates maintains a non-zero probability of receiving each evaluated therapy. Because conditional exchangeability remains inherently untestable in observational settings, unmeasured confounding can easily induce substantial bias into downstream predictive assessments.

When investigators suspect that unmeasured confounding severely undermines the observational exchangeability assumption, the research team outlines a transportability analysis as an alternative estimation framework. This strategy leverages the randomized treatment assignment of a clinical trial to identify the relationship between treatments and outcomes, transporting those causal insights to the target population represented by the observational registry. The transportability pathway operates under assumptions A1* through A5*, where consistency, trial-level conditional exchangeability, and trial treatment positivity are reinforced by the trial’s experimental design. However, transporting trial results across cohort boundaries introduces assumption A4*, which demands conditional exchangeability between the randomized trial super-population and the observational population given measured baseline covariates. Furthermore, assumption A5* establishes positivity of being in the target population, mandating that every covariate configuration present in the observational database has a non-zero probability of appearing within the randomized trial cohort. The authors highlight that transportability becomes mathematically invalid whenever the observational registry spans clinical characteristics excluded by trial eligibility criteria, unless investigators resort to untestable extrapolation models.

Both the observational and transportability paradigms yield concrete two-step estimation procedures for evaluating generic loss functions—such as mean squared error, absolute loss, or Brier scores—associated with an arbitrary machine learning model. In the observational pipeline, researchers first train an outcome model estimating the expected loss conditional on baseline covariates among observational participants who actually received the specified treatment strategy. That fitted model is subsequently applied to predict individual counterfactual losses across all observational participants, and the resulting predictions are averaged over the registry cohort. The transportability workflow mirrors this sequence but trains its initial outcome model strictly within the randomized trial arm assigned to the treatment of interest. Those trial-derived parameters are then projected directly onto the covariate patterns of the observational database to compute the final performance estimate. An essential advantage of the transportability approach is that it requires neither outcome measurements nor treatment records from the observational registry, rendering it functional when real-world registries suffer from substantial outcome misclassification or incomplete follow-up.

Beyond evaluating observational and transportability pipelines in isolation, the investigators derive theoretical properties for a joint analysis framework that pools both data sources when all underlying identifiability assumptions hold simultaneously. Under this unified framework, the expected outcome conditional on baseline covariates and treatment assignment remains identical across both the randomized trial and the observational study. This mathematical equivalence permits researchers to fit a single outcome model on the pooled multi-source dataset before marginalizing the resulting predictions over the target observational population. By incorporating individual participant data from both the trial and the registry, the joint analysis strategy achieves greater statistical efficiency than either standalone approach. The authors demonstrate mathematically and through asymptotic variance derivations that the joint estimator exhibits an asymptotic variance that is less than or equal to the variance of both the observational and transportability estimators. This theoretical variance reduction translates directly into narrower confidence intervals and improved precision when assessing machine learning risk.

To illustrate how selectively violating foundational assumptions affects model performance, the researchers executed a controlled simulation evaluating four distinct data-generating scenarios involving a continuous outcome, binary treatment, measured covariate, and unmeasured confounder. In the first scenario, where all conditional exchangeability conditions were preserved, both the observational and transportability estimators demonstrated unbiased performance across five hundred simulation iterations, aligning closely at the origin. In the second scenario, unmeasured factors introduced additional outcome variance within the observational cohort, violating cross-population exchangeability while preserving treatment exchangeability; this led the transportability estimator to severely underestimate true mean squared error, whereas the observational estimator remained completely unbiased. In the third scenario, an unmeasured confounder directly influenced treatment decisions in the observational data, violating observational exchangeability; consequently, the observational estimator exhibited pronounced bias while the transportability estimator preserved unbiased accuracy. Finally, when both exchangeability assumptions were violated simultaneously, both estimation approaches suffered severe bias, demonstrating how latent confounding and cross-population heterogeneity distort algorithmic validation.

Because the core assumptions underpinning both analytical strategies cannot be verified using empirical data alone, the authors examine benchmarking as a vital methodological practice for evaluating estimation integrity. Benchmarking involves systematically comparing performance estimates derived from the randomized trial via transportability against estimates generated directly from the observational registry. When both independent analytical pathways yield concordant risk estimates and overlapping confidence intervals, researchers gain empirical confidence that neither unmeasured treatment confounding nor unmeasured cross-population differences are materially compromising the results. However, the authors explicitly emphasize that successful benchmarking cannot mathematically guarantee validity, because both estimators could theoretically suffer from undetected biases of similar magnitude and direction. Conversely, when benchmarking reveals meaningful discordance between the two estimates, investigators are definitively alerted that at least one core exchangeability assumption has failed, precluding uncritical pooling of the datasets and signaling the necessity of formal sensitivity analyses or careful domain-specific re-evaluation.

The research team applied their complete methodological toolkit to empirical clinical data from the Coronary Artery Surgery Study, a comprehensive historical cohort that enrolled participants between 1975 and 1979 with follow-up extending through 1996. The study evaluated the clinical efficacy of coronary artery bypass grafting surgery combined with medical therapy versus medical therapy alone among patients presenting with significant coronary artery disease and reduced left ventricular ejection fraction. Within the study design, eligible participants could choose to enroll in a randomized clinical trial or, upon declining randomization, enter a parallel observational registry. Analyzing a complete-case sample of 1,686 participants, the authors trained a random forest model on a fifty-percent training partition to predict ten-year patient mortality. They then evaluated the model’s Brier risk under counterfactual treatment strategies of universal surgery versus universal medical therapy across the remaining observational cohort, implementing outcome modeling, inverse probability weighting, and doubly robust estimators across transportability, observational, and joint configurations.

The empirical results from the Coronary Artery Surgery Study illustrated both the practical utility of benchmarking and the efficiency advantages of multi-source joint estimation. For the counterfactual surgical intervention strategy, all point estimates and ninety-five percent bootstrap confidence intervals aligned closely across observational, transportability, and joint analyses, providing robust benchmarking concordance while the joint estimator delivered visibly narrower confidence bounds. Under the counterfactual medical therapy strategy, however, the researchers detected minor discordance between observational and transportability estimates, suggesting a potential mild violation of unmeasured confounding or population exchangeability—possibly driven by unrecorded clinical factors such as prior peripheral vascular interventions. The authors concluded that while their estimation framework provides model-agnostic tools compatible with both complex machine learning architectures and classical regression, deploying these methods requires rigorous domain knowledge to evaluate untestable assumptions. Future methodological priorities highlighted in the paper include developing formal sensitivity analyses, accommodating non-adherence and missing data, and extending transportability theory to settings without individual participant data.

Subject of Research: Counterfactual prediction and performance evaluation for machine learning models using multi-source clinical data

Article Title: Counterfactual prediction from machine learning models: transportability and joint analysis for model development and evaluation using multi-source data

Article References: Voter, S. C., Dahabreh, I. J., Boyer, C. B., Rahbar, H., Kontos, D., & Steingrimsson, J. A. (2025). Counterfactual prediction from machine learning models: transportability and joint analysis for model development and evaluation using multi-source data. Diagnostic and Prognostic Research, 9(1), Article 22. https://doi.org/10.1186/s41512-025-00201-y

Image Credits: AI Generated

DOI: 10.1186/s41512-025-00201-y

Keywords: counterfactual prediction, machine learning evaluation, transportability analysis, joint estimation, observational health data, randomized controlled trials, conditional exchangeability, unmeasured confounding, treatment assignment, Brier score, Coronary Artery Surgery Study, doubly robust estimation, positivity assumption, statistical efficiency, benchmarking, asymptotic variance, outcome model estimators