Predicted outcomes as virtual controls: five conditions for valid treatment effect estimates

When a new treatment arrives in the clinic before randomized trial evidence is available, researchers sometimes turn to an intuitive shortcut: take the patients who received the new therapy, and use a statistical model built on older data to predict what would have happened to them under the old, standard treatment. The average gap between what actually happened and what the model predicted is then taken as an estimate of the treatment’s benefit. A new methodological paper in Diagnostic and Prognostic Research, published on 24 July 2026, formalizes this strategy for the first time within the potential outcomes framework of causal inference and spells out exactly when the shortcut is trustworthy and when it can silently mislead.

The work, led by Lotta M. Meijerink and colleagues including Artuur M. Leeuwenberg, Jungyeon Choi, Bas B. L. Penning de Vries, Karel G. M. Moons and Ewoud Schuit, was motivated by a practice that has grown familiar in radiotherapy research, where it is known as model-based clinical evaluation, but has also appeared in oncology, cardiology and mental health research. Although the approach can be recognized as a specific case of standardization, a method of virtual controls or a g-method, the authors note that the theory and the conditions required for unbiased treatment effect estimation had not previously been formally described. Their stated objective was to close that gap.

The quantity the method targets is the average treatment effect among the treated, or ATT: the average difference between the outcome a patient would experience under the new treatment and the outcome that same patient would have experienced under the standard treatment, restricted to the patients who actually received the new therapy. To estimate it, researchers fit a prediction model in a historical or external cohort in which every patient received the standard treatment. The model uses patient characteristics, such as age or comorbidities, and treatment plan variables describing how the standard therapy would have been delivered, such as planned radiation doses to specific organs. That fitted model is then applied to the patients treated with the new therapy, generating counterfactual predictions of their outcomes under the standard treatment, and the average of observed minus predicted outcomes yields the ATT estimate.

The authors identify five sufficient conditions for this estimate to be valid: transportability, ignorability of treatment assignment, consistency, positivity, and correct model specification. They stress that weaker conditions could suffice in some settings, but that this set offers an intuitive framework for understanding what can go wrong. Crucially, most of these conditions are largely untestable, because the counterfactual outcomes being predicted are never observed. The paper therefore also describes how empirical evidence, such as comparing predicted and observed outcomes in related samples, can raise confidence that the assumptions are plausible.

The first condition, transportability, requires that the risk of the outcome under the standard treatment, given the covariates in the model, is the same in the pre-introduction and post-introduction populations. It applies only to outcomes under the standard treatment, since those are the only ones the model predicts. Violations arise when a risk factor is left out of the model and its prevalence differs between populations, for example if smoking raises the outcome risk but is more common in the historical cohort, causing the model to overestimate risk when applied to the newer population. Concurrent changes in treatment protocols, or differences in dose optimization strategies that shift radiation doses to organs not captured by the model, are other examples the authors give.

The second condition, ignorability of treatment assignment, demands that within the post-introduction population, the potential outcome under standard treatment is independent of which treatment the patient actually received, given the covariates. This is what allows the model to make valid counterfactual predictions in a non-random subset of patients, namely those selected for the new therapy; if selection had been random, the condition would hold automatically. Violations occur when access to the new treatment depends on characteristics that also influence the outcome independently of the modeled covariates, such as socioeconomic status tied to insurance coverage, or when clinicians assign treatment based on patient-reported quality of life that predicts the outcome but is absent from the model.

Consistency, the third condition, requires that each patient’s observed outcome matches their potential outcome under the treatment they actually received, which holds when the treatment has a single well-defined meaning: the specific equipment, clinician or workflow used to deliver it should not change the outcome. The authors offer examples including insulin delivered by pump versus manual injection, outcomes that depend heavily on a clinician’s skill in surgery or radiotherapy planning, and older versus newer radiation machines that approximate the same planned dose with different accuracy. Positivity, the fourth condition, requires that every combination of covariate values that could occur among patients receiving the new treatment could also occur in the population used to develop the model; otherwise the model must extrapolate, essentially guessing outcomes for patients unlike anyone seen before. The authors distinguish structural violations from stochastic ones, where rare combinations are simply missing from the available data by chance, which does not invalidate theoretical identifiability but can still cause practical problems.

The fifth condition is correct model specification: the outcome model must have the right functional form, or be flexible enough, to capture the conditional probability of the outcome given the included covariates. A nonlinear age effect modeled linearly, or an unmodeled interaction between tumor location and sex, would violate it. Notably, the authors point out that this condition concerns the relationship between included covariates and the outcome, not whether additional covariates would improve the model. Because the estimand is an average across the treated population, some specification errors may partially average out, but the authors still recommend sensitivity analyses examining how different modeling choices affect the resulting estimate.

To make the framework concrete, the authors illustrate every condition with a case study from Dutch radiotherapy, based on synthetic data modeled on a real-world setting. The question was the benefit of proton therapy over conventional photon therapy, specifically volumetric modulated arc therapy, in reducing dysphagia, difficulty swallowing of grade 2 or higher, at six months after radiotherapy in patients with head and neck cancer. A historical sample of 750 patients treated with photons between 2007 and 2017 was used to fit a logistic regression model incorporating baseline dysphagia status, tumor location, and four planned dose parameters from the photon plan: the mean planned doses to the superior, middle and inferior pharyngeal constrictor muscles and the oral cavity. After proton therapy was introduced in 2018, reimbursement was restricted to patients expected to benefit substantially, determined by a model-based selection procedure that compared each patient’s photon and proton plans and selected those whose expected benefit exceeded ten percent. Ninety-three patients were treated with protons under this scheme, while 207 received photons in the same period. In the synthetic worked example, the average observed outcome under proton therapy was 0.25, the average predicted counterfactual risk under photon therapy was 0.47, giving an ATT estimate of minus 0.22 with a 95 percent confidence interval of minus 0.31 to minus 0.13, a benefit the authors note is expected given that patients were selected on expected benefit.

The case study also shows how each condition can be probed. Transportability could be threatened if dose optimization improved over time, lowering doses to organs not in the model and thereby inflating the apparent benefit of protons. Ignorability is delicate here because selection depended on planned proton doses, which were unavailable when the model was developed; unmeasured variables such as N-stage, which the authors note is not in the model, could correlate with both achievable dose reduction and dysphagia risk, potentially underestimating the benefit. Positivity was expected to hold because both samples came from the same hospital. For empirical support, the authors used calibration-in-the-large: among post-introduction patients who received photon therapy, the difference between observed and predicted outcomes was 0.0 with a 95 percent confidence interval of minus 0.05 to 0.05, providing no evidence of violated conditions, though the authors caution that good calibration in this group does not guarantee unbiased estimates for the proton-treated subgroup. A second validation, applying the model directly to proton-treated patients under the assumption that dose-outcome relationships carry over, yielded a calibration-in-the-large of minus 0.05, suggesting predicted risks about five percent higher than observed, which under strong additional assumptions would imply some overestimation of the treatment effect, with uncertainty remaining high due to limited data. The authors conclude that when the prediction model predicts well in relevant subpopulations the approach can yield unbiased estimates, but that many sources of bias exist, and they recommend systematic consideration of all required conditions, informed by domain expertise, alongside empirical evidence wherever possible. They also acknowledge limitations: the work focuses on identification rather than how violations translate into bias in practice, addresses binary outcomes and the ATT specifically, and uses a simplified case study that omits real-world complexities such as missing data, competing risks and measurement error.

Subject of Research: Conditions for valid estimation of treatment effects by comparing observed outcomes with model-predicted counterfactual outcomes under standard treatment

Article Title: Treatment effect estimation by comparing observed and predicted outcomes: conditions for valid inference and practical illustration

Article References: Meijerink, L. M., Leeuwenberg, A. M., Choi, J., Penning de Vries, B. B. L., Langendijk, J. A., van Loon, J. G. M., Nout, R. A., Moons, K. G. M., & Schuit, E. (2026). Treatment effect estimation by comparing observed and predicted outcomes: conditions for valid inference and practical illustration. Diagnostic and Prognostic Research, 10(1), Article 20. https://doi.org/10.1186/s41512-026-00233-y

Image Credits: AI Generated

DOI: 10.1186/s41512-026-00233-y

Keywords: treatment effect estimation, causal inference, potential outcomes, average treatment effect among the treated, counterfactual prediction, proton therapy, photon therapy, dysphagia, head and neck cancer, radiotherapy, prediction model validation, transportability