What Speech AI Sees in the Brain: Interpretable Features Explain Audio Model Success

Speech foundation models such as Whisper, HuBERT, and WavLM are increasingly hailed as powerful computational models of human speech perception, yet their representations remain black boxes, leaving unclear what drives their alignment with brain responses. A new study by Riki Shimizu, Richard J. Antonello, Chandan Singh, and Nima Mesgarani tackles this question by decomposing the models’ success into interpretable components.

The team built linear encoding models from six interpretable feature families—mel-spectrograms, Gabor filter bank features, speech presence, phonetic features, syntactic features, and semantic features—alongside contextualized embeddings from the three state-of-the-art speech foundation models. They quantified electrocorticography response variance shared between feature classes.

Variance-partitioning analyses revealed several key insights. First, the speech foundation models’ alignment with the brain can be mostly accounted for by their ability to learn and encode simple interpretable speech features. Second, the models exhibit a systematic trade-off between encoding brain-relevant low-level and high-level features across layers.

Third, the models learn brain-relevant semantics that cannot be accounted for by lower-level speech features, with this capacity increasing with model size and context length.

The authors propose a principled approach to build more interpretable, accurate, and efficient encoding models of the brain by augmenting speech foundation model embeddings with interpretable features. The study, published in PLOS Computational Biology, turns an interpretability problem into a quantitative accounting exercise: decompose the black box, attribute its neural predictive power to known speech features, and identify the semantic surplus that remains.

Subject of Research: How interpretable speech features explain and enhance the brain encoding performance of speech foundation models

Article Title: Interpretable embeddings of speech explain and enhance the brain encoding performance of audio models

Article References: Shimizu, R., Antonello, R. J., Singh, C., & Mesgarani, N. (2026). Interpretable embeddings of speech explain and enhance the brain encoding performance of audio models. PLOS Computational Biology, 22(10), e1014906. https://doi.org/10.1371/journal.pcbi.1014906

Image Credits: AI Generated

DOI: 10.1371/journal.pcbi.1014906

Keywords: speech foundation models, brain encoding, electrocorticography, variance partitioning, interpretable features, Whisper, HuBERT, WavLM, semantics, computational neuroscience, Interpretable, embeddings