A multicenter investigation across South Korea has demonstrated that pairing patient voice recordings with traditional cognitive screening scores significantly improves deep learning classification of dementia. Conducted by researchers including Hee Won Yang, So Yeon Jeon, and Jeong Lan Kim, the study evaluated how different diagnostic data streams collected during routine Mini-Mental State Examination sessions contribute to distinguishing between cognitively normal status, mild cognitive impairment, and dementia. By training neural network architectures on patient audio and drawings alongside clinical evaluations, the team established that acoustic features add measurable predictive value over clinical scores alone.
The multicenter study gathered diagnostic and clinical records from 1,091 participants evaluated across five university hospitals in South Korea. The underlying clinical objective was to develop and validate an integrated multimodal deep learning framework while dissecting the relative utility of audio responses, pencil-and-paper drawings, and standard clinical metrics. To process these heterogeneous data streams systematically, the researchers transformed voice recordings captured during cognitive screening into mel-frequency cepstral coefficient spectrograms and digitized pentagon drawing tasks into standard grayscale images.
To evaluate these modalities computationally, the investigators applied the DenseNet121 deep neural network architecture separately to the spectrograms and grayscale drawing images. Concurrently, participant demographic details and clinical evaluation scores were processed through a multilayer perceptron. The diagnostic performance across single-modality and multimodal configurations was comprehensively measured using classification accuracy alongside the area under the receiver operating characteristic curve, with statistical validation performed using five-fold cross-validation.
In the challenging three-class evaluation distinguishing between cognitively normal individuals, mild cognitive impairment, and dementia, the full multimodal deep learning model achieved an overall accuracy of 67.9 percent, with a 95 percent confidence interval spanning 65.9 to 70.0 percent. The corresponding area under the receiver operating characteristic curve reached 0.84, with a 95 percent confidence interval of 0.82 to 0.85. Analytical breakdown revealed that remaining classification errors occurred predominantly between individuals who were cognitively normal and those diagnosed with mild cognitive impairment.
For binary classification distinguishing cognitively normal individuals from patients with dementia, the multimodal deep learning framework achieved robust discriminatory power. In this diagnostic setting, the model attained an accuracy of 90.1 percent, with a 95 percent confidence interval ranging from 88.4 to 91.8 percent, paired with an area under the receiver operating characteristic curve of 0.97 and a 95 percent confidence interval between 0.95 and 0.97. These performance metrics underscored the efficacy of integrating distinct data channels for identifying established dementia cases.
A critical technical finding from the comparative model evaluations was that voice-based architectures consistently outperformed drawing-only models across evaluation benchmarks. When acoustic representations were combined directly with clinical evaluation measures, the resulting models demonstrated higher overall diagnostic discrimination than models trained exclusively on clinical measures alone. This demonstrates that automated acoustic processing captures relevant cognitive indicators that supplement standard clinical measures.
In contrast to the clear gains provided by voice spectrograms, static pentagon drawings contributed only a modest diagnostic benefit to the multimodal network. The inclusion of drawing data provided minor improvements primarily in the area under the curve when discriminating mild cognitive impairment, while offering limited general predictive enhancement across other screening comparisons. Consequently, static visual representations from the copying test yielded less distinctive utility than acoustic features captured during patient responses.
The researchers determined that voice data obtained during routine clinical screening provides valuable information complementary to standard Mini-Mental State Examination scores. Combining automated voice analysis with standard screening assessments offers a practical and scalable approach to cognitive screening, particularly for detecting dementia, while highlighting the clear translational potential of voice-based digital biomarkers in clinical environments.
Subject of Research: Multimodal deep learning for cognitive screening using voice and drawing data
Article Title: What do voice and drawings recorded during the MMSE add to its score? A multicenter multimodal deep learning study of dementia screening
Article References: Yang, H. W., Jeon, S. Y., Ahn, K., Yoo, S., Byeon, G., Moon, S. W., Park, J. H., Kwak, K. P., Cho, M., & Kim, J. L. (2026). What do voice and drawings recorded during the MMSE add to its score? A multicenter multimodal deep learning study of dementia screening. GeroScience. https://doi.org/10.1007/s11357-026-02647-4
Image Credits: AI Generated
DOI: 10.1007/s11357-026-02647-4
Keywords: dementia screening, deep learning, multimodal machine learning, voice biomarkers, Mini-Mental State Examination, mild cognitive impairment, DenseNet121, spectrogram analysis, cognitive assessment, digital biomarkers, clinical neuroinformatics, mel-frequency cepstral coefficients, medical artificial intelligence, geriatric screening, receiver operating characteristic, cross-validation
News Source: Scienmag.
