MEMOIR-VLM: an AI that reads brain scans and answers questions about Alzheimer’s disease

Alzheimer’s disease affects an estimated 55 million people worldwide, and that number is projected to nearly double by 2050. Yet roughly three-quarters of people with dementia have never received a formal diagnosis, and in low- and middle-income countries as many as 90 percent of cases go unrecognized. A team at the Mark and Mary Stevens Neuroimaging and Informatics Institute at the University of Southern California believes artificial intelligence can help close that gap—and they have built a system that does something most clinical AI cannot: it classifies disease stage, predicts cognitive severity, and then explains itself in plain English.

The system, called MEMOIR-VLM (Multimodal Encoder with Missing-modality and Open-ended Inference for Retrieval), is described in a new paper in Frontiers in Computational Neuroscience. It is a two-stage vision-language framework trained and evaluated on 2,363 subjects from the Alzheimer’s Disease Neuroimaging Initiative (ADNI), one of the largest publicly available brain-imaging cohorts for dementia research. The first stage is a deep learning encoder that fuses three very different kinds of data: T1-weighted structural MRI, which captures brain atrophy; diffusion tensor imaging (DTI) fractional anisotropy maps, which quantify white matter microstructure; and structured clinical scores such as the MMSE, ADAS-11, ADAS-13, MoCA, and APOE genotype. The second stage extends that frozen encoder into a retrieval-augmented generation pipeline that can answer open-ended questions about a brain scan in natural language.

The technical heart of the system is its missing-modality handling. In real clinical settings, data is almost never complete: DTI requires longer scan times, is sensitive to motion artifacts, and is often absent from standard imaging protocols. Most multimodal deep learning models either impute the missing data with generative models—introducing hallucination risk—or restrict training to subjects with complete data, sacrificing sample size. MEMOIR-VLM takes a third route. Each modality embedding passes through a binary masking gate, and during training these gates are sampled stochastically according to empirical availability rates (10 percent dropout for T1w, 30 percent for DTI-FA, 5 percent for clinical scores). At inference time, gating is deterministic and reflects exactly which modalities are available for a given patient. A learnable pooling query then attends over the available modality tokens through eight-head cross-attention, producing a single fused 512-dimensional representation. One set of weights therefore operates on any of the seven non-empty modality subsets, from a single modality to the full set, without imputation and without a separate model per subset.

Each imaging modality is processed by its own 3D ResNet-18 encoder, trained with separate weights so that anatomical and microstructural patterns are captured without interference. Clinical features flow through a multi-layer perceptron that also embeds APOE genotype as a categorical input. Critically, the model excludes target variables—diagnosis, sex, age, and amyloid status—from the clinical input space to prevent label leakage. After CLIP-style InfoNCE contrastive pretraining, which aligns embeddings of the same subject across modalities, the encoder is fine-tuned for five simultaneous prediction targets: three-way diagnosis (cognitively normal, MCI, or dementia), binary diagnosis, Clinical Dementia Rating Sum of Boxes (CDR-SB) severity, age, and biological sex.

The results are striking. On held-out ADNI test data, with CDR-SB excluded from the inputs to remove a cognitive-score leak, the encoder achieved 91.3 percent balanced accuracy for distinguishing cognitively normal subjects from those with dementia, and 68.2 percent balanced accuracy for the harder three-class staging task (macro-F1 = 0.681). Age was predicted with a mean absolute error of 5.96 years. Per-class analysis revealed that dementia is the easiest class to identify (86 percent recall), while MCI is the hardest (55 percent recall)—a natural consequence of MCI’s transitional, heterogeneous nature. Even so, extreme misclassifications were nearly absent: only 3 of 474 subjects were confused between cognitively normal and dementia, indicating the model learns a well-ordered representation of disease progression where errors occur almost exclusively between adjacent stages.

The second stage is where the system becomes conversational. A lightweight text encoder, all-MiniLM-L6-v2, is first pretrained with masked language modeling on 26,889 clinical sentences, then aligned to the frozen image embeddings through a small projection layer trained with symmetric InfoNCE loss. Two FAISS indices—one over fused image embeddings and one over projected text embeddings—enable exact cosine-similarity search over the 1,889 training subjects. For each query, FAISS retrieves the top-50 most similar historical cases; a cross-encoder reranks the top-20 candidates on caption similarity; and the top-5 are passed as context to a Mistral-7B-Instruct language model loaded in 4-bit quantization, which generates structured answers or free-form clinical descriptions. Importantly, the language model never sees raw brain images. Its visual understanding is grounded entirely in the encoder’s embedding space, ensuring interpretability and traceable evidence chains.

The team also conducted an unusually candid audit of their own pipeline. When retrieved captions contained the ground-truth diagnosis field, the VQA layer appeared to reach 94.7 percent accuracy—but the researchers showed this was a label-exposure artifact: the cross-encoder matched diagnosis strings between captions, and the language model simply copied stated labels. When the diagnosis field was withheld from all captions, no evaluated language model exceeded a retrieval-only k-NN majority vote (0.673), which itself closely matched the encoder’s own three-class accuracy (0.682). The team therefore recommends a deployment configuration in which the diagnostic label is produced by the encoder, while the retrieval-augmented language layer supplies the natural-language rationale, the five most similar comparator cases, and an auditable evidence chain. The generated text itself showed strong semantic fidelity, with BERTScore of 0.894 and SBERT cosine similarity of 0.811 for structured question answering.

Perhaps the most compelling evidence for the missing-modality design came from external validation. The identical ADNI-trained weights were run zero-shot on OASIS-3, a cohort of 1,048 subjects with no processed diffusion data at all. With the DTI branch masked, the model retained 78.7 percent balanced accuracy for distinguishing cognitively normal from impaired subjects (AUC = 0.889)—a result a model requiring complete modality sets, or relying on generative imputation, could not achieve without additional machinery and hallucination risk. Three-class performance degraded to 0.524, driven chiefly by the small MCI class, indicating that the transferable signal lies at the extremes of the disease spectrum.

The modality ablation also clarified what imaging is actually for. Clinical scores dominated diagnosis classification, and a simple two-layer MLP on the same clinical features was statistically indistinguishable from the full multimodal model on both diagnostic tasks. But imaging added a statistically supported signal on top of cognition for three-class diagnosis when DTI-FA was integrated alongside clinical context (clinical-only 0.666 vs. 0.701 with DTI added; paired difference +0.034, 95% CI excluding zero), and imaging was the modality that mattered most for age estimation, where the imaging-only configuration outperformed clinical-only by 0.63 years MAE. The authors are explicit that they do not claim the model extracts staging information a well-administered cognitive battery does not already carry; its value lies in robustness when cognition is absent, in targets cognition cannot address, and in severity gradation.

With an end-to-end inference latency of 4 to 6 seconds per patient on a single consumer-grade GPU requiring roughly 8 GB of VRAM, MEMOIR-VLM could be deployed without high-performance computing infrastructure. The authors suggest the encoder could serve as a scalable first-stage screening component to flag individuals requiring specialist referral, particularly valuable in settings where neuropsychological assessment is unavailable. Beyond Alzheimer’s disease, the architectural principles—missing-modality cross-attention fusion, multi-task clinical prediction, and retrieval-grounded natural-language reasoning—may extend to radiologic AI more generally, and even to brain-computer interface systems that routinely integrate heterogeneous neural signals with variable availability across subjects and sessions. The convergence of multimodal encoders, large language models, and retrieval-augmented generation demonstrated here represents a promising blueprint for the next generation of interpretable neuroimaging AI.

Subject of Research: A multimodal vision-language model for Alzheimer's disease classification and question answering

Article Title: MEMOIR-VLM—a multimodal vision-language model for Alzheimer's disease classification and question answering

Article References: Gidwani, S. H., Chattopadhyay, T., Thomopoulos, S. I., & Thompson, P. M. (2026). MEMOIR-VLM—a multimodal vision-language model for Alzheimer's disease classification and question answering. Frontiers in Computational Neuroscience, 20, Article 1902258. https://doi.org/10.3389/fncom.2026.1902258

Image Credits: AI Generated

DOI: 10.3389/fncom.2026.1902258

Keywords: Alzheimer's disease, vision-language model, deep learning, retrieval-augmented generation, brain MRI, diffusion tensor imaging, large language models, medical AI, ADNI, visual question answering, MEMOIR-VLM, multimodal