Drift metrics predict how synthetic clinical text fails on real patient notes

Synthetic clinical notes generated by large language models have been promoted as a privacy-friendly way to train medical text-processing systems, but a new study shows that the gap between machine-made and real clinical writing can be measured — and that this measurement predicts how badly de-identification models will stumble when moved from synthetic training data to genuine patient records.

De-identification — the automatic removal of protected health information (PHI) such as names, addresses, contact details and dates from clinical text — is what allows hospitals to share electronic health records for research while respecting privacy laws like HIPAA in the United States and GDPR in Europe. Public training data for this task remain scarce, however, because manual de-identification is labor-intensive and clinical notes cannot easily be shared across institutions.

One increasingly popular workaround is to have large language models fabricate synthetic clinical notes that can be annotated consistently and generated in any language without exposing a single real patient. Many healthcare institutions with limited computing resources still rely on smaller pretrained language models for de-identification, since only compact models can be trained and deployed fully on-premise as regulations require. Synthetic data, which can be produced off-premise, looks like an ideal fuel for these small models. The catch, documented repeatedly in prior literature, is that models trained on synthetic text often lose noticeable accuracy when applied to real clinical documents, because institution-specific writing styles, entity distributions and annotation conventions are not fully captured by the generators.

To quantify that loss, the researchers built a synthetic corpus from scratch covering four PHI categories: Address, Contact, Date and Person. They used an ensemble of five generator models — Meditron3 Qwen2.5 7B, Qwen3 8B, Llama 3.1 8B Instruct, Ministral 8B Instruct and MedGemma-4B Instruct — supervised by a slightly larger judge model, Qwen3 14B, which scored every generated sample on a one-to-five quality scale. Generation ran for 30 rounds with 250 samples per round. The first round was strictly zero-shot; after that, ten highly scored samples were drawn from the growing pool as few-shot examples for the next round, an iterative self-improvement loop that required no manually written seed examples. A judge threshold of 3.0 filtered the data, a cap of ten occurrences per entity string prevented repetition, and a negative-sample ratio of 0.7 controlled class balance. The resulting notes were short, roughly two to three sentences and about 57 to 61 tokens each, with entity counts ranging from 41,000 for Person up to 110,000 for Address, provided in training sizes of 1,000, 10,000 and 20,000 documents plus a fixed 2,000-document test split.

For comparison, the team drew on two real-world resources and one additional synthetic one. I2B2, a benchmark of long-form clinical notes averaging roughly 60 sentences and 859 tokens per document, served as the key evaluation target. MultiLeg, a legal-domain dataset with very short documents of about 53 tokens, and Ai4Privacy, the largest dataset in the mix with a vocabulary exceeding 55,000 types and multiple sublabels for all four PHI categories, were used for both training and evaluation — though Ai4Privacy is itself synthetic, and its construction methods have not been released.

The researchers fine-tuned two compact pretrained language models, BioClinicalBERT and DeBERTa v3 base, for token classification on BIO-formatted PHI spans, training one model per entity type with two random seeds per configuration, five epochs, a batch size of 64 and a learning rate of 2×10⁻⁵. Evaluation used strict entity-level matching to compute precision, recall and F1 score. The cross-dataset results confirmed a familiar pattern: every model performed best on its own training distribution. Scaling Ai4Privacy from 1,000 to 20,000 training samples improved both in-distribution and out-of-distribution scores on I2B2, yet Ai4Privacy models transferred only moderately to the legal-domain MultiLeg data for Address and Contact, exposing a domain and style gap. The home-grown synthetic corpora scored near-perfectly on their own synthetic tests and were competitive on I2B2 for Date and Person, but remained weak on clinical-style Address, which appears harder to generalize when generated in isolation.

The heart of the study lies in its drift analysis. The team computed 24 metrics for each entity-specific train-test pair, spanning three families: surface statistics such as Jensen–Shannon divergence over unigrams and bigrams, vocabulary Jaccard similarity, lexical diversity measures including MTLD and Yule’s K, distinct n-gram rates, a four-gram self-repetition rate to detect template overuse, Flesch–Kincaid readability differences and GPT-2 perplexity; structural indicators such as line-break density, key-value-style sentences, bullet openings, ALL-CAPS tokens and special characters; and semantic metrics computed in embedding space, including maximum mean discrepancy with an RBF kernel, Fréchet distance between Gaussian summaries, cluster-based Jensen–Shannon drift and the Euclidean distance between embedding means. Each metric was ranked by its absolute Pearson correlation with ΔF1, the performance gap between the best real-data-trained model and the mean synthetic-trained model, where higher values signal greater degradation.

The strongest predictors turned out to be the distributional ones. Jensen–Shannon divergence over unigrams and bigrams both correlated at r = 0.54 with F1 degradation, followed by MTLD difference at r = 0.53, perplexity difference at r = 0.52 and Flesch–Kincaid difference at r = 0.51. Representation-level metrics — cluster drift, embedding mean distance and maximum mean discrepancy — landed at roughly 0.49 to 0.50, confirming that the shift is visible in embedding space as well. Vocabulary overlap showed an inverted correlation of −0.47, which simply reflects that higher overlap means lower drift. Strikingly, style-based features such as capitalization, punctuation and special characters showed no clear predictive patterns, suggesting that transfer difficulty is governed by token-distributional and semantic-coverage gaps rather than by the superficial stylistic quirks that make synthetic text feel artificial to human readers. Values around 0.5 indicate a moderate positive association between higher drift and greater performance loss, and the heatmap analysis revealed a sharp separation between in-distribution and out-of-distribution evaluations for most metrics.

Three ablation studies sharpened the picture. Filtering synthetic data by judge score benefited Person and Date most when only the highest-scoring samples (5.0) were kept, while Address and Contact changed little. Mixing real and synthetic data proved the most actionable finding: models trained with 20 to 60 percent synthetic data achieved higher out-of-distribution F1 than models trained only on real data for all four entity types, with the largest relative gains for Date, whereas performance declined or stagnated above 80 percent synthetic content. In other words, synthetic data works best as an additive signal to real corpora rather than as a full substitute. Comparing the five generator models showed largely similar out-of-distribution scores, with only small differences for Person and Date. In the ablation drift analysis, correlations grew stronger than in the main cross-domain analysis, led by vocabulary overlap at r = −0.78, cluster drift at r = 0.72, unigram Jensen–Shannon divergence at r = 0.71 and bigram divergence at r = 0.69, indicating that lexical coverage metrics become especially discriminative when distributional differences are confined to pipeline variations rather than sweeping domain shifts.

The authors are careful about scope. This is an exploratory study, and its observations are specific to their generation pipeline, their datasets, their harmonized label space and the English language; they call for validation on other synthetic data generation approaches, additional institutions, languages and tasks. Still, the practical implication is concrete: because drift metrics require no access to labeled target data, they can serve as a quality-control screen that estimates transfer risk before any model training begins, complementing existing quality filtering in synthetic-data pipelines.

Subject of Research: Correlating distributional drift between synthetic and real clinical text with PHI de-identification model performance

Article Title: Measuring the gap: correlating synthetic-to-real drift with PHI de-identification performance

Article References: Cornelius, J., & Rinaldi, F. (2026). Measuring the gap: correlating synthetic-to-real drift with PHI de-identification performance. Genomics & Informatics, 24(1), Article 10. https://doi.org/10.1186/s44342-026-00072-9

Image Credits: AI Generated

DOI: 10.1186/s44342-026-00072-9

Keywords: synthetic clinical text, large language models, de-identification, protected health information, distribution drift, out-of-distribution generalization, clinical NLP, Jensen-Shannon divergence, embedding metrics, privacy, electronic health records, fine-tuning