Medical artificial intelligence is advancing on a foundation that is easy to overlook: carefully labeled images. A new comprehensive review has mapped 140 benchmark datasets used to train and evaluate deep-learning systems for medical image segmentation, revealing both the extraordinary breadth of available resources and the persistent weaknesses that can limit clinical reliability. The survey, published in Artificial Intelligence Review by Anzhi Wang, Chengbang Yang, Xu Zhang, Xi Yang, Weihua Ou and colleagues, organizes datasets spanning seven major anatomical regions and more than 60 tissue and lesion categories. Its central message is that medical segmentation algorithms can only be as dependable as the data used to build and test them—and that today’s benchmark landscape remains fragmented, uneven and difficult to compare across studies.
Segmentation is the process of assigning a label to individual pixels or three-dimensional voxels in a medical scan. Instead of merely identifying that a tumor, organ or blood vessel is present, a segmentation model traces its precise boundaries. This information can be used to calculate tumor volume, measure organ shape, plan radiation treatment, guide surgery or monitor disease progression. Deep-learning systems typically learn this task from images paired with expert annotations, often called masks. During training, a neural network adjusts millions of internal parameters to reduce the difference between its predicted mask and the reference mask. The quality, consistency and diversity of those reference masks therefore directly influence what the model learns—and whether it can operate safely on images from a different hospital, scanner or patient population.
The researchers divide the benchmark ecosystem into five broad task scenarios. The first concerns segmentation of a single organ or tissue, such as the liver, brain, heart or retinal structures. The second focuses on lesion regions, including tumors and other abnormal areas. The third covers multiple organs or tissues within the same image, while the fourth addresses several lesion types or regions. The fifth combines multi-organ or multi-tissue segmentation with lesion detection and delineation. This classification matters because the computational difficulty changes substantially between tasks. A model that isolates one relatively uniform organ may face a very different problem from a system required to identify several organs, distinguish normal anatomy from lesions and preserve boundaries where tissues have similar intensity values.
The review’s dataset inventory spans multiple imaging modalities, including the familiar technologies of computed tomography and magnetic resonance imaging as well as other forms of clinical imaging. Each modality presents a different technical challenge. CT measures X-ray attenuation and often provides strong contrast for bone, lung and some abdominal structures, but exposes patients to ionizing radiation. MRI produces images through magnetic fields and radiofrequency signals, offering excellent soft-tissue contrast but often varying considerably with scanner strength, acquisition sequence and protocol. Ultrasound can capture images in real time, yet its appearance is affected by speckle noise, probe angle and operator technique. These differences create what machine-learning researchers call domain shift: a model trained on one distribution of images may lose accuracy when the image statistics change.
Benchmark datasets are intended to make algorithmic comparisons fair, but the survey shows why that goal is difficult to achieve. Datasets differ in image resolution, field of view, patient demographics, disease prevalence, annotation policy and the number of experts involved in labeling. Some contain only a small number of highly curated cases; others offer larger collections with less uniform annotations. A mask drawn by one radiologist may not match one drawn by another, particularly when a tumor boundary is indistinct or an organ is partially obscured. Even the definition of the target can vary. One study may label the visible tumor core, another the entire abnormal region and a third include surrounding tissue suspected of microscopic invasion. A model can therefore appear to perform differently not because its architecture changed, but because the task itself was defined differently.
Deep-learning segmentation is commonly assessed with overlap metrics such as the Dice similarity coefficient and intersection over union. The Dice score compares the overlap between the predicted region and the ground-truth region, doubling the shared area and dividing it by the total area of both regions. A score of one represents perfect agreement, whereas zero indicates no overlap. Intersection over union divides the shared area by the combined area and is more strongly penalized when the predicted and reference regions differ. Researchers may also report Hausdorff distance, which measures the greatest or near-greatest boundary discrepancy, and average surface distance, which captures how far the predicted contour lies from the reference contour. These metrics measure different aspects of performance: a high overlap score does not necessarily guarantee clinically acceptable boundaries, especially for small lesions where a few misplaced pixels can have major consequences.
The survey also highlights the problem of data leakage and overly optimistic evaluation. If images from the same patient, examination session or institution appear in both training and test sets, a model may exploit repeated visual patterns rather than learn general anatomical principles. Randomly splitting images is not always sufficient, because multiple slices from a single three-dimensional scan are highly correlated. More robust evaluation separates data at the patient level and, where possible, tests models on external datasets collected using different equipment or protocols. Without such safeguards, benchmark results may exaggerate how well an algorithm would function in clinical practice. A system that achieves impressive performance on a familiar dataset can still fail when confronted with motion artifacts, unusual anatomy, postoperative changes or a scanner it has never encountered.
The authors argue that the field’s next phase must move beyond simply collecting more images. Larger datasets are valuable, but scale alone cannot resolve incomplete demographic representation, inconsistent labeling or weak documentation. Future resources should record relevant information about patients, acquisition settings, annotation procedures and the provenance of each image while protecting privacy. Standardized labeling protocols and multiple expert annotations could make it possible to quantify uncertainty rather than treating one mask as an unquestionable truth. Privacy-preserving approaches, including de-identification, federated learning and carefully governed data-sharing frameworks, may help institutions collaborate without transferring raw patient records. Synthetic images and data augmentation can expand training diversity, but they must be validated carefully so that artificial examples do not introduce unrealistic anatomy or erase clinically important variation.
The dataset map could become a practical navigation tool for researchers developing the next generation of medical AI. By bringing together resources for organs, tissues, lesions and combined segmentation tasks, it offers a way to identify gaps, select more appropriate benchmarks and design evaluations that better reflect real-world use. The authors have also assembled a related online collection of medical segmentation datasets, intended to support continued exploration. The larger scientific significance is a shift in emphasis: progress should not be measured only by whether a new neural-network architecture raises a score on a familiar benchmark, but by whether it remains accurate, transparent and useful across hospitals, populations and imaging conditions. In medical imaging, the most powerful algorithm is not necessarily the one that wins a single leaderboard. It is the one whose performance survives contact with the messy biological and technological diversity of actual patients.
Subject of Research: Benchmark datasets for deep learning-based medical image segmentation
Article Title: A comprehensive review of benchmark datasets for deep learning-based medical image segmentation
Article References: Wang, A., Yang, C., Zhang, X. et al. “A comprehensive review of benchmark datasets for deep learning-based medical image segmentation.” Artificial Intelligence Review (2026). Original research article
Image Credits: AI Generated
DOI: 10.1007/s10462-026-11662-y
Keywords: medical image segmentation, deep learning, benchmark datasets, artificial intelligence, lesion segmentation, organ segmentation, medical imaging, dataset bias
Tags: anatomical region-specific datasetsapplications of segmentation in tumor and organ analysisbenchmark datasets for AI in healthcarechallenges in medical image dataset standardizationclinical reliability of AI-based segmentationcomprehensive review of medical image datasetsdataset fragmentation in medical image analysisdeep learning in medical imagingevaluation of medical segmentation algorithmsmedical image annotation and labelingmedical image segmentation datasetstissue and lesion classification datasets
