At a prototyping workshop convened in Berlin from November 6 to November 8, 2024, researchers from the Optimal Treatment for Patients with Solid Tumours in Europe Through Artificial Intelligence consortium presented an integrated computational framework combining clinical records, genomic profiling, radiological imaging, and digital pathology. The collaborative initiative was designed to address core clinical challenges in oncology, focusing on the epidemiological burden of breast and lung cancers across the European Union. Breast cancer remains among the most common malignancies globally, exceeding 2.3 million annual diagnoses worldwide and representing roughly 29 percent of female cancer diagnoses within the European Union, where member states experience notable disparities in mortality. Concurrently, lung cancer accounts for approximately 470,000 new diagnoses each year in the European Union, constituting 11.9 percent of all new cancer cases and leading cancer mortality by causing 20.4 percent of cancer-related deaths. While non-small cell lung cancer represents about 85 percent of cases, small cell lung cancer forms an aggressive subset accounting for roughly 15 percent. To confront these challenges, the consortium organized analytical workflows around four primary research objectives: assessing comprehensive genomic profiling in metastatic breast cancer, examining tumor heterogeneity to predict therapeutic responses, developing non-invasive computational alternatives to traditional invasive tissue biopsies, and analyzing optimal therapeutic sequences for stage I and II small cell lung cancer.
To unlock critical diagnostic and staging details stored in unstructured clinical documentation, the initial workshop module developed a rule-based natural language processing pipeline to extract oncological features from narrative electronic health records. Analyzing records from the Breast Cancer Now Biobank at the Barts Cancer Institute, investigators examined clinical data mapped to the Observational Medical Outcomes Partnership Common Data Model linked with records from Barts Health NHS Trust. The study cohort focused on breast cancer patients with audited records covering at least five years of follow-up or survival until death, aligning with the temporal window of highest recurrence risk. Investigators filtered electronic radiology reports by examination date to remove procedures conducted prior to primary diagnosis and by imaging modality to retain relevant scans such as mammograms, computed tomography, magnetic resonance imaging, and ultrasound examinations while removing unrelated records like dental X-rays. Automated preprocessing removed structured header variables, including examination timestamps, modality tags, and radiologist identities, before resolving sentence boundaries across heterogeneous headings such as clinical indication, findings, and conclusions. Textual normalization expanded abbreviations and clipped terms like mets, converted text to lowercase, stripped punctuation and stop words, and applied look-ahead negation logic. Using the WordNet lexical corpus within the Natural Language Toolkit in Python alongside Pandas, terms linked to the body part synset through hypernym matching were flagged to identify candidate metastatic locations. Evaluating consecutive lemma sequences and bigrams confirmed the viability of automated text mining against validated biobank labels.
A second workflow integrated clinical demographics, somatic mutation data, and transcriptomic profiling using the Cancer Genome Atlas Breast Invasive Carcinoma dataset to characterize tumor heterogeneity and explore therapeutic vulnerabilities. Somatic variants cataloged in Mutation Annotation Format files were paired with whole-transcriptome expression matrices and an external curated pharmacogenomic reference detailing known drug targets and mutation-specific therapy responses. Analytical pipelines implemented in R utilizing packages including plotly, Maftools, ggalluvial, Genefu, and xCell revealed distinct demographic patterns, such as Black patients presenting with breast cancer approximately four years earlier than White patients within the cohort. Mutational mapping identified TP53 and PIK3CA as dominant genetic alterations, occurring at frequencies of 32 percent and 34 percent, respectively. Stratification focused on key PIK3CA hotspot missense mutations, specifically H1047R, E545K, and E542K, which carry direct therapeutic relevance in estrogen receptor-positive and human epidermal growth factor receptor 2-positive disease. Integrating transcriptomic profiles enabled sample subtyping into classical molecular categories and cellular deconvolution of the tumor microenvironment. This transcriptomic analysis revealed significant variations in immune cell infiltration across subtypes, with basal-like and HER2-enriched tumors displaying significantly elevated immune scores relative to luminal subtypes. These immune infiltration patterns suggest differential susceptibility to immunotherapeutic interventions, aligning with spatial digital pathology analyses and setting the stage for future longitudinal circulating tumor DNA monitoring.
Addressing the objective of developing non-invasive diagnostic tools, a third session established a two-stage deep learning pipeline to evaluate digital mammograms for malignancy and intrinsic molecular subtypes. Utilizing craniocaudal and mediolateral oblique views from the Chinese Mammography Database archived within The Cancer Imaging Archive, the team standardized images through spatial resizing, pixel intensity normalization, and Gaussian filtering for noise suppression. The initial detection stage employed a You Only Look Once version 9 architecture trained to isolate suspicious regions of interest containing abnormal tissue, minimizing computational processing burdens while refining classification focus. The localized regions of interest were subsequently transferred to a ResNet-50 convolutional neural network pretrained on ImageNet for feature extraction and classification. To prevent overfitting and enhance generalizability across limited samples, training incorporated extensive data augmentations, including random horizontal and vertical flipping, rotations, and spatial scaling. The network underwent optimization over 100 epochs using mini-batch stochastic gradient descent with a learning rate of 0.0001, a batch size of 32, early stopping based on validation loss, and checkpointing to preserve the top-performing model weights.
The deep learning mammography pipeline was evaluated on binary discrimination between benign and malignant lesions, alongside multiclass categorization into Luminal A, Luminal B, HER2-enriched, and Triple-Negative molecular subtypes. On the binary classification task, the model achieved an overall accuracy of 74.84 percent, a precision of 73.14 percent, and a sensitivity recall of 67.14 percent. For the more challenging task of directly predicting four discrete molecular subtypes from mammographic features alone, the network attained an accuracy of 58 percent when tested on a restricted evaluation subset. To interpret the underlying deep representations and substantiate algorithmic reasoning, the researchers implemented Gradient-weighted Class Activation Mapping alongside t-Distributed Stochastic Neighbor Embedding. Class activation heatmaps confirmed that predictive weights localized directly to clinically meaningful radiographic lesions, displaying intense focal activation over tumor masses while blue and yellow zones marked background tissue. Concurrently, dimensionality reduction via t-SNE demonstrated distinct clustering between benign and malignant cases in latent space, with further separation of molecular subclusters among malignant cases. These explainability techniques provide critical transparency, offering objective visual verification to reinforce clinical confidence when evaluating non-invasive radiological models designed to augment conventional biopsy workflows.
The fourth workshop module investigated spatial tumor architecture through digital histopathology, processing whole slide images from 24 patients within the TCGA-BRCA cohort. Expert pathologists annotated tissue regions using QuPath software across eight distinct morphological categories: Tumor, Stroma, Normal Ducts, Immune Cells, Adipose Tissue, Necrosis, Complex, and Background. High-resolution slides were partitioned into non-overlapping patches measuring 256 by 256 pixels, strictly partitioned at the patient level into training (70 percent), validation (10 percent), and held-out testing (15 percent) sets to prevent cross-contamination of biological signals. Implemented in PyTorch, a ResNet-50 deep convolutional network was trained using cross-entropy loss and the Adam optimizer, with rotation and flipping augmentations applied to balance minority histological classes. Model performance was evaluated on the independent test set using accuracy, precision, recall, and F1-score metrics, supplemented by confusion matrices, receiver operating characteristic curves, and class-specific area under the curve measurements. Slide-level quantification calculated exact area proportions for each tissue class, while patch-level predictions were projected as color-coded spatial overlays directly onto original whole slide images to visualize microarchitectural patterns and immune infiltration landscapes.
Recognizing that grouping all malignant epithelium into a single generalized tumor class obscured intricate cellular morphology, the digital pathology team contrasted the standard ResNet-50 pipeline against a foundation model developed by Bayer. Built upon a vision transformer architecture, this foundation model was pretrained in an unsupervised self-supervised manner across 64 million histopathological image patches, acquiring complex representations of biological tissue without task-specific supervisory labels. A linear classification layer fitted atop the vision transformer representations revealed fine-grained morphological subclusters within the malignant tissue class when visualized using t-Distributed Stochastic Neighbor Embedding. These latent subclusters mapped directly to subtle variations in tumor grading, architectural patterns, and cellular differentiation, demonstrating that foundation models capture extensive biological heterogeneity that conventional supervised convolutional networks compress. When paired with Gradient-weighted Class Activation Mapping to verify that spatial attention aligned with tumor cell borders and stromal margins, this approach established a framework for decoding spatial heterogeneity. Future iterations aim to integrate this digital pathology workflow with transcriptomic datasets, such as targeted RNA sequencing from the PALOMA-3 trial, to dissect the interplay between spatial tissue composition and clinical drug resistance.
The fifth workshop module addressed early-stage small cell lung cancer by establishing a federated observational research framework using the Observational Medical Outcomes Partnership Common Data Model. To investigate treatment patterns and clinical outcomes for this aggressive malignancy across decentralized healthcare settings, the consortium deployed distributed analytics across 11 distinct health data sources spanning eight countries: Belgium, Estonia, Germany, Italy, Romania, Spain, the United Kingdom, and the United States. Participating sources included primary care networks such as CPRD GOLD, SIDIAP, THIN, and IQVIA Disease Analyzer Germany, specialized oncology electronic medical records including IQVIA OncoEMR, and population genomics resources from the Estonian Biobank. Using the PhenotypeR analytical package developed within the OPTIMA initiative, investigators defined and executed diagnostic cohort algorithms across both broad and narrow small cell lung cancer definitions. Concept set diagnostics systematically evaluated patient counts, concept frequencies, index event distributions, orphan diagnostic codes, age and sex demographics, incidence, and prevalence. This federated design utilized the central data platform hosted at Helmholtz-Zentrum Dresden-Rossendorf for importable datasets while dispatching standardized analytical scripts to remote servers behind institutional firewalls, enabling local data owners to return aggregate summaries without exposing patient-level records.
To study limited-stage small cell lung cancer cohorts, researchers established strict inclusion criteria requiring patients to be at least 18 years of age with at least one continuous year of observation prior to diagnosis. Patients were required to have confirmed small cell lung cancer without antecedent primary malignancies, excepting non-melanoma skin cancer, alongside documented TNM staging confirming stage I or II tumor extent recorded within a window extending from one month prior to two months following diagnosis. Clinical interventions were mapped according to National Comprehensive Cancer Network 2025 guidelines, encompassing surgical resections such as lobectomy, lung excision, or sleeve resection, systemic chemotherapy regimens containing platinum compounds and etoposide such as carboplatin or cisplatin, and external beam radiation techniques including stereotactic, conformal, and intensity-modulated radiotherapy. Primary clinical endpoints comprised overall survival and disease progression, structured around Response Evaluation Criteria in Solid Tumors criteria assessed across predefined intervals of three months, six months, nine months, 12 months, 18 months, 24 months, five years, and 10 years. Implemented using OHDSI ATLAS and an interactive R Shiny application adapted from OHDSI CohortDiagnostics, the phenotyping algorithm confirmed that small cell lung cancer represented approximately 15 percent of all lung cancer diagnoses, with early-stage disease comprising only about 5 percent of small cell cases. Despite this rarity, the federated network projected an aggregate sample size of approximately 7,000 eligible patients across partner databases.
The integrated findings from the prototyping workshop highlighted significant technical and epidemiological barriers that must be resolved to translate multimodal computational pipelines into oncological practice. Cross-database comparisons within the lung cancer network identified substantial discrepancies in real-world recording practices, particularly concerning TNM staging, where clinical staging prefixes, pathological notations, or composite scores were frequently missing or discordant due to OMOP mapping omissions or charting gaps. Furthermore, the deep learning mammography pipeline demonstrated that while binary discrimination of malignant lesions reached reliable performance, predicting granular molecular subtypes directly from imaging remains constrained by sample size limitations, achieving 58 percent accuracy. The OPTIMA consortium concluded that continuing development will prioritize expanding training datasets, evaluating longitudinal liquid biopsies and circulating tumor DNA, and integrating natural language processing with imaging algorithms to monitor treatment response. Realizing the full clinical potential of multimodal artificial intelligence requires refining multi-institutional data curation, resolving discrepancies across observational databases, and establishing standardized federated frameworks that uphold data privacy while linking deep clinical, genomic, imaging, and histopathological resources.
Subject of Research: Multimodal artificial intelligence workflows and distributed clinical data analytics for breast and lung cancer management
Article Title: Multimodal AI-driven analysis in breast and lung cancer: insights from the OPTIMA prototyping workshop
Article References: Abdollahyan, M., Gadaleta, E., James, L. G. E., Thorn, G. J., Corz, J. C., Singh, V., Hettich, M., Schneider, L., Maiques, O., Hijazy, A., Golozar, A., Rivas, J. G., Cornford, P., Nicolleti, R., Abbott, T., Tan, E. H., Newby, D., Papanastasiou, G., Bussmann, M., … the PIONEER consortium and the OPTIMA consortium (2026). Multimodal AI-driven analysis in breast and lung cancer: insights from the OPTIMA prototyping workshop. BMC Proceedings, 20(S15), Article 14. https://doi.org/10.1186/s12919-026-00370-8
Image Credits: AI Generated
DOI: 10.1186/s12919-026-00370-8
Keywords: Artificial Intelligence, Breast Cancer, Lung Cancer, Multimodal Analysis, Natural Language Processing, Digital Pathology, Deep Learning, OMOP Common Data Model, Bioinformatics, Pharmacogenomics, Multimodal, AI-driven
News Source: Nathaniel Bowman. (November 10, 2026). OPTIMA Prototyping Workshop Demonstrates Multimodal AI Workflows for Breast and Lung Cancer. Scienmag.



