OCR-Extracted MRS Metabolites Aid Brain Lesion Classification

Researchers from Vellore Institute of Technology in Chennai, India, have developed a multimodal machine learning pipeline that classifies brain lesions by fusing structural magnetic resonance imaging (MRI) radiomics with metabolic data recovered from magnetic resonance spectroscopy (MRS) screenshots. The study, published in Frontiers in Neurology, demonstrates that optical character recognition (OCR) can extract quantitative metabolite values directly from rendered clinical spectrum images, a step rarely attempted in prior automated classification work. In a cohort of 55 patients, the multimodal pipeline achieved an accuracy of 82.0% under a bias-corrected nested cross-validation protocol, outperforming a non-OCR baseline that relied solely on MRI radiomics and MRS screenshot texture.

The central challenge addressed by the team is the diagnostic ambiguity of structural MRI alone. Tumors, infections, and demyelinating disorders can present with structurally indistinguishable morphologies on standard MRI sequences, particularly in early-stage or atypical cases. While MRI excels at delineating lesion shape and spatial extent, it cannot directly characterize the underlying tissue biochemistry. MRS addresses this gap by quantifying key metabolites, such as choline (Cho), which reflects membrane turnover, and N-acetyl aspartate (NAA), which signals neuronal integrity. Despite this utility, MRS data is often archived as rasterized screenshots rather than raw spectral files, making it inaccessible to direct machine reading.

To overcome this barrier, the authors designed a custom OCR pipeline using Tesseract. The process involves selecting the axial slice of maximum variance from the MRS screenshot, normalizing the image, and applying multi-region cropping with Lanczos upsampling. Lenient regular expression parsing with character substitutions was then used to recover metabolite values, including Cho, Cr, NAA, and their ratios. This automated extraction allows for the integration of metabolic features into machine learning models without requiring access to raw spectral signal files, which are seldom available in routine clinical workflows.

The study utilized a publicly available dataset comprising 55 patient folders, each containing NIfTI volumes for three MRI sequences (T2-FLAIR, T1-weighted, and T1 with contrast) alongside a single MRS screenshot. The cohort included 17 patients with non-neoplastic lesions and 38 with neoplastic lesions. Diagnostic labels were derived from consensus expert diagnoses recorded in the source dataset, which incorporated clinical presentation, multi-sequence MRI, and MRS review by treating radiologists. The authors explicitly note that independent histopathological confirmation via biopsy was not systematically available for every case, introducing a degree of circularity when using MRS-derived features to predict labels that originally relied on MRS input.

MRI preprocessing involved denoising, intensity normalization, and lesion segmentation using Dynamic Image Thresholding (DIT). From the segmented regions, the team extracted 113 raw radiomic features, including first-order intensity statistics, Grey-Level Co-occurrence Matrix (GLCM) texture, morphological shape, and contrast-enhancement ratios. After removing zero-variance columns and excluding wavelet features that added dimensional noise, the final feature set for the multimodal model comprised 71 features. This set included the MRI radiomics, demographic data, MRS screenshot texture, and the nine OCR-extracted metabolite features.

The classification protocol employed a leakage-controlled Leave-One-Out Cross-Validation (LOOCV) design to ensure that all learned transformations, including imputation, feature selection, and normalization, were fitted only on the training data within each fold. The primary performance estimate was derived from a nested LOOCV protocol, where the classifier family and feature count were selected using an inner loop on training data alone. This approach yielded a bias-corrected accuracy of 82.0%. In contrast, the best fixed configuration, a Random Forest model with k=3 features, reached an exploratory accuracy of 85.5% (47/55 correct) with an Area Under the Curve (AUC) of 0.82. The authors emphasize that the 85.5% figure is optimistically biased because the configuration was selected on the same data it was scored on.

Comparison with a non-OCR baseline, which used 62 features excluding the OCR-extracted metabolites, revealed a clear advantage for the multimodal approach. The best configuration for the baseline reached 78.2% accuracy, indicating a gain of 7.3 percentage points for the multimodal model at the best-configuration level. The nested estimate exceeded the baseline by 3.8 points. Statistical testing supported these findings: a 1,000-permutation test showed that the best-configuration accuracy significantly exceeded the random-label mean of 55.0%, and McNemar’s exact test returned a p-value of 0.05, indicating a borderline but consistent improvement over the baseline.

Explainability analysis using SHAP (SHapley Additive exPlanations) values confirmed that the model’s decisions were driven primarily by the OCR-recovered metabolites. Log-transformed Cho and NAA emerged as the strongest discriminators, with ANOVA F-scores of 12.89 and 16.01, respectively. These metabolites accounted for 78.0% of the total mean absolute SHAP magnitude, while MRI-derived features contributed the remaining 22.0%. This distribution aligns with established clinical biochemistry, where elevated choline is a hallmark of neoplastic proliferation and reduced NAA indicates neuronal loss or displacement.

Validation of the OCR pipeline against manual transcription of all 55 patients showed high reliability. Detection rates varied by metabolite, with Cho recovered in 43 of 55 patients and Cho/Cr in only 24 of 55. However, a parallel run using manually transcribed values yielded an accuracy of 86.6%, only 1.1 percentage points higher than the 85.5% achieved with OCR-recovered values. This narrow gap demonstrates that the automated pipeline recovers essentially all of the discriminative signal available in the screenshots, despite residual extraction noise.

The authors highlight several limitations, primarily the small sample size of 55 patients, which precludes the use of conventional held-out test splits and necessitates the use of LOOCV. The circularity of using MRS features to predict labels that were originally informed by MRS review means the results should be interpreted as evidence of reproducing expert diagnostic judgment rather than independent pathological ground truth. Furthermore, the OCR step remains susceptible to errors in poorly rendered screenshots, though sanity filters and imputation strategies mitigate these issues. The study serves as a proof-of-concept for integrating MRS into automated classification pipelines in settings where raw spectral data are unavailable, suggesting that future work should focus on larger cohorts and quantitative spectral fitting methods.

To determine how much each individual data stream contributed to the classification gain, the researchers evaluated four isolated feature configurations using a fixed Random Forest classifier with five features under the same cross-validation protocol. Strict MRI radiomics and patient demographics alone, comprising forty-six raw features without any spectroscopy data, reached an accuracy of 78.2 percent. The nine raw optical character recognition metabolite features alone achieved a comparable 77.3 percent accuracy. However, combining strict MRI radiomics with the extracted metabolite features elevated the accuracy to 85.5 percent, representing a jump of more than seven percentage points over either modality in isolation. Meanwhile, models trained solely on the twenty raw texture features derived from the spectroscopy screenshot images performed worst at 71.2 percent accuracy. This sharp contrast confirmed that the performance boost stemmed from genuine biochemical values rather than incidental image patterns or rendering artifacts in the archived screenshots.

Before adopting handcrafted radiomics, the investigators explored end-to-end deep learning methods by training several convolutional neural network architectures directly on MRI slices, including a custom lightweight network, TinyVGG, and ResNet variants. Across all trials, validation accuracy remained trapped between 55 and 65 percent with severe fold-to-fold instability, demonstrating that hundreds of thousands of learnable parameters could not generalize on a clinical sample of fifty-five patients despite extensive data augmentation. The team likewise evaluated principal component analysis to reduce the 109 retained features, noting that eight components captured 80 percent of the total variance and eighteen components captured 95 percent. Nevertheless, dimension-reduced classifiers consistently underperformed unreduced models across all tests. This shortfall occurred because the directions of greatest variance in a 109-dimensional feature space do not necessarily correspond to the directions of maximum class separation, and reliable covariance estimation remains unachievable with fifty-five subjects.

Performance varied noticeably across diagnostic subtypes in the best fixed model, reflecting distinct underlying metabolic profiles. High-grade tumors were detected with the greatest accuracy at 93.3 percent, or twenty-eight out of thirty cases, whereas low-grade tumors reached 75.0 percent accuracy, or six out of eight cases, and non-neoplastic lesions achieved 76.5 percent, or thirteen out of seventeen cases. This separation mirrors the biochemical distributions recorded in the cohort: neoplastic lesions exhibited roughly 3.2 times higher median choline levels at 60,800 units compared to 12,896 units in non-neoplastic lesions, alongside more than double the median choline-to-NAA ratio. Because aggressive malignancies undergo pronounced cellular proliferation, their marked metabolic departure from healthy tissue makes them far easier for the multimodal classifier to distinguish than benign or low-grade abnormalities that share overlapping biochemical boundaries.

To verify that data preprocessing did not introduce subtle biases, the authors also examined whether missing metabolite values clustered systematically by diagnostic category. Fisher exact tests comparing data availability across non-neoplastic and neoplastic cohorts yielded significance values above 0.20 for all metabolites, proving that missingness occurred independently of diagnosis and preventing median imputation from leaking diagnostic clues. Agreement between optical character recognition and manual reading showed high consistency across readable ranges, with Bland-Altman analyses detecting no systematic measurement bias. Looking ahead, the researchers emphasized that future work must validate the pipeline against independent biopsy-confirmed cohorts, longitudinal progression, and patient survival endpoints.

Subject of Research: Multimodal brain lesion classification approach through MRI radiomics and OCR-recovered MRS metabolites with explainable AI

Article Title: Multimodal brain lesion classification approach through MRI radiomics and OCR-recovered MRS metabolites with explainable AI

Article References: Balaji, M., Fathima, A., Khadeeja, N., & Maheswari, N. (2026). Multimodal brain lesion classification approach through MRI radiomics and OCR-recovered MRS metabolites with explainable AI. Frontiers in Neurology, 17, 1978957. https://doi.org/10.3389/fneur.2026.1978957

Image Credits: AI Generated

DOI: 10.3389/fneur.2026.1978957

Keywords: Multimodal, brain, lesion, classification, approach, radiomics, OCR-recovered, metabolites, explainable