Machine learning maps England’s costliest patients into nine distinct groups

A tiny fraction of patients consumes an enormous share of the National Health Service’s budget, and a new machine learning analysis of more than ten million English adults has now revealed just how varied that population really is. Researchers at the University of Manchester and Imperial College London set out to answer a deceptively simple question: who exactly are the patients whose care costs the most, and can they be grouped in ways that help the health system plan better? Their findings, published in BMC Medicine, show that the top one percent of patients by cost accounted for 26.8 percent of total healthcare spending in the study population, yet these patients are far from a uniform block of elderly, hospital-dependent individuals. Instead, unsupervised clustering algorithms identified nine distinct groups, each with its own combination of diagnoses, age profile, deprivation level and pattern of service use.

The scale of the underlying dataset is one of the study’s most striking features. The team examined primary and secondary care costs for the financial year 2018/19 among 10,119,490 adult patients registered with general practices contributing to the Clinical Practice Research Datalink and linked to Hospital Episode Statistics, a cohort representing roughly 23 percent of the adult population of England. Because the data capture both general practice contacts and hospital activity, the researchers could compute a genuinely comprehensive picture of each patient’s annual cost to the system, valued in 2018 prices using established unit cost references. High-cost patients were defined as those falling in the top one percent of total care costs, a threshold widely used in health economics because spending is so heavily concentrated at the extreme of the distribution.

The numbers illustrate that concentration vividly. High-cost patients had average annual costs of £17,485, with a median of £14,609 and an interquartile range running from £12,028 to £19,633. The rest of the population, by contrast, averaged just £653 per year, with a median of only £103 and an interquartile range of £14 to £352. In other words, the typical high-cost patient cost the system well over a hundred times as much as the typical patient, and even the cheapest quarter of the high-cost group outspent the most expensive quarter of everyone else by a wide margin. Understanding who these patients are, the authors argue, is therefore central to any serious attempt at resource allocation, case management and health system planning.

Methodologically, the study is a careful exercise in unsupervised learning, the branch of machine learning that searches for structure in data without any predefined labels. The researchers clustered patients on the basis of diagnoses, age, sex, deprivation measured by the Index of Multiple Deprivation, and ethnicity. They compared multiple algorithms, including approaches built on multiple correspondence analysis and t-distributed stochastic neighbour embedding for dimensionality reduction, before settling on hierarchical clustering as the optimal method. That choice was not arbitrary: the team evaluated candidate solutions using established validity and stability indices, including the Davies–Bouldin Index, the Calinski–Harabasz index and the Adjusted Rand index, which reward clusters whose members are internally similar, well separated from other clusters, and reproducible under resampling.

The winning solution produced nine clusters with clearly differentiated multimorbidity and sociodemographic profiles. Mean age across the clusters ranged from 56 to 79 years, a reminder that high costs are not confined to the very old. Mean annual costs per cluster ranged between £15,792 and £19,107, with confidence intervals indicating that these differences were estimated precisely. The spread of clinical pictures was equally wide. Several core profiles recurred across different clustering approaches and across subgroup analyses, suggesting they represent genuine and robust archetypes within the high-cost population rather than statistical artefacts of a single algorithm.

Among those recurring archetypes, four stand out. Younger people with liver disease and mental health conditions formed one recognisable group, a combination that clinicians will recognise as associated with complex social circumstances and frequent emergency attendance. Patients with nodal metastases, meaning cancers that have spread to lymph nodes, formed another, as did patients with prostate cancer complicated by nodal metastases. The fourth recurring profile comprised older people with cardiovascular disease and dementia, a group whose costs typically accumulate through repeated admissions, long hospital stays and heavy reliance on both primary and secondary care. That the same profiles emerged repeatedly, including in analyses restricted to in-year decedents and to patients with the highest unplanned care costs, strengthens the case that these are stable, clinically meaningful segments.

The study also stratified its analyses by age, examining age-stratified subgroups alongside the whole population. This matters because the drivers of high cost plausibly differ across the life course: a 58-year-old with alcoholic liver disease and depression uses services very differently from an 80-year-old with heart failure and cognitive decline. By confirming that the core clinical profiles persisted across these stratifications, the researchers provide evidence that segmentation is not simply a proxy for age. The consistency of the clusters across methods and subgroups was assessed formally, and the authors report that the hierarchical solution performed best on the validity indices they applied, giving the nine-group taxonomy a firmer empirical footing than ad hoc clinical categorisations.

Why does this matter for policy? Population health management, the approach increasingly adopted by integrated care systems in England, depends on being able to identify groups of patients whose needs are similar enough to be addressed by a common service model. If the top one percent of patients were homogeneous, a single intensive case management programme might suffice. The new analysis shows the opposite: a younger cohort dominated by liver disease and mental illness needs addiction, psychiatric and social care integration, while an older cohort with dementia and cardiovascular disease needs falls prevention, medication review and coordinated discharge planning. Designing services for a blended average of these groups risks serving none of them well. Cluster-based segmentation, the authors suggest, could inform the targeting of case management programmes, the allocation of budgets and longer-term workforce and capacity planning.

The research also carries implications for how health systems think about equity. By incorporating deprivation and ethnicity into the clustering alongside diagnoses, the analysis acknowledges that cost is shaped by social context, not just by disease burden. Patients in deprived areas often present later, have more multimorbidity at younger ages, and generate higher unplanned care costs, so a segmentation that ignores socioeconomic characteristics would systematically misdescribe the population. The study was approved by the Clinical Practice Research Datalink’s independent scientific advisory committee and used anonymised patient data under ethical approval from the Health Research Authority, and the authors declare no competing interests. Funding came from the Health Foundation and the National Institute for Health and Care Research, with the funders having no role in study design, analysis or the decision to publish.

Limitations and next steps remain. The analysis reflects a single financial year, 2018/19, before the disruptions of the pandemic reshaped waiting lists and care patterns, and cost data capture NHS spending rather than the full social care picture. Unsupervised clustering, however carefully validated, identifies statistical structure rather than causal mechanisms, so the clusters describe who high-cost patients are rather than why their costs are high. Even so, the study offers one of the most comprehensive segmentations of a national high-cost population attempted anywhere, drawing on a cohort of over ten million linked primary and secondary care records. Its central message is likely to travel well beyond England: the costliest patients are a heterogeneous collection of recognisable clinical archetypes, and health systems that learn to see them as distinct groups, rather than as an undifferentiated burden, will be better placed to design care that is both more effective and more humane.

Subject of Research: Machine learning segmentation of high-cost patients in the English NHS

Article Title: Segmentation of high-cost patients in England: insights from a machine learning analysis

Article References: Segmentation of high-cost patients in England: insights from a machine learning analysis. (n.d.). https://doi.org/10.1186/s12916-026-05233-9

Image Credits: AI Generated

DOI: 10.1186/s12916-026-05233-9

Keywords: high-cost patients, machine learning, hierarchical clustering, multimorbidity, NHS, health economics, population health management, CPRD, Hospital Episode Statistics, patient segmentation, healthcare costs, England