Hippocampus Represents Combined Audiovisual Experiences Beyond Simple Sums

A new study published in PLOS Biology provides evidence that the hippocampus, a brain structure traditionally associated with visual memory, is actively involved in representing both unimodal and multimodal sensory experiences. While much of the existing research on the hippocampus has focused on visual stimuli, such as isolated pictures or words, this work addresses a significant gap in understanding how the brain processes the rich, dynamic, and multisensory nature of everyday life. The study, led by researchers including Raccah and colleagues, utilized functional magnetic resonance imaging to examine how the hippocampus encodes information when visual and auditory streams are presented together, separately, or in mismatched combinations. The findings suggest that the hippocampus does not merely process visual data but integrates complex sensory inputs to form coherent representations of real-world events.

The disconnect between controlled laboratory studies and real-world experiences has long been a challenge in cognitive neuroscience. Traditional studies often use impoverished stimuli, such as single words or static images, which fail to capture the complexity of daily interactions. Although recent efforts have used more realistic stimuli like television shows and movies to bridge this gap, it remained unclear which specific features of multimodal stimuli are represented by the hippocampus. Furthermore, the manner in which the combination of these features alters neural representations was not well understood. This new research attempts to disentangle these representations by selectively decomposing visual-only or auditory-only information from the same stimulus and presenting them in congruent or incongruent audiovisual pairings.

To investigate these questions, participants underwent brain imaging while repeatedly viewing ten short video clips, each lasting six seconds. These clips depicted real-world experiences, such as a ping-pong game or animals making noises, including birds chirping and wolves howling. The experimental design included two main conditions: unimodal presentations and multimodal presentations. In the unimodal condition, participants experienced either only the visual content or only the auditory content of the clips. For example, they might watch a wolf crane its neck and look up at the sky without hearing the howl, or hear the howl without seeing the wolf. This approach allowed the researchers to isolate the neural contributions of each sensory modality.

In the multimodal condition, the audio and visual components were carefully remixed across all video clips. Sometimes, the audio and visual information came from the same clip, creating congruent pairings where the sensory streams matched. Other times, the audio and visual information was recombined from different clips, resulting in incongruent pairings. For instance, participants might see a wolf looking up at the sky while hearing the sound of a typewriter. This manipulation was crucial for testing whether the hippocampus prioritized unimodal information or multimodal integration, and how it handled mismatches between sensory inputs.

The researchers analyzed the fMRI data using two distinct methods: univariate analyses and multivariate analyses. Univariate analyses focused on aggregated levels of activity within the hippocampus, providing a broad view of overall activation. In contrast, multivariate analyses examined distributed activity profiles, offering a more nuanced perspective on how specific patterns of neural activity corresponded to different stimulus types. This dual approach allowed the study to reveal differences that might be obscured by looking at average activity levels alone.

The univariate analyses revealed that the hippocampus reflected either visual-only information or combined audiovisual information, but not auditory information alone. This initial finding suggested a potential bias toward visual or integrated processing at the level of overall activity. However, the multivariate analyses uncovered a more complex picture. The distributed activity profiles indicated that the hippocampus represented audio-only, visual-only, and their combination. Importantly, only a coherent link between the audio and visual information led to an enhancement of these representations. This suggests that the hippocampus is sensitive to the congruence of sensory inputs, not just their presence.

Further analysis of the congruent audiovisual stimuli showed that the hippocampus was representing the gestalt of the audio and visual information, meaning the whole experience above and beyond the simple combination of its component parts. The representations of congruent audiovisual stimuli were stronger than those for auditory-only or visual-only movies. Additionally, these representations explained unique variation in the measured hippocampal responses that could not be attributed to either unimodal presentation alone. This indicates that the hippocampus performs a form of integration that creates a distinct neural signature for coherent multimodal experiences.

The study also examined whether the location along the long axis of the hippocampus mattered for these distributed patterns. The posterior hippocampus represented the movies in a way that was specific to the visual modality. In contrast, the anterior hippocampus represented the movies in a way that generalized across modalities. For example, the audio-only and visual-only versions of the same movie produced more similar activity profiles in the anterior hippocampus than unimodal presentations of different movies. This finding is consistent with existing evidence suggesting a functional gradient along the hippocampus, where anterior regions represent information more abstractly and posterior regions represent it in a more detailed or specific manner.

These findings are significant because they help explain how the hippocampus supports memory for the kinds of dynamic, multimodal experiences that characterize everyday life. By using different analytical approaches, the study reveals the nuanced differences between aggregating activity within a region and understanding the variation within that region across anatomical delineations and distributed activity profiles. The results highlight the importance of considering both the location within the hippocampus and the pattern of activity when studying memory-related processes.

The hippocampus has historically been appreciated for its role in supporting memory, but more recent characterizations have highlighted its close ties to the visual system and a broad role in cognitive operations ranging from attention and language to perception and prediction. This work further suggests a need for broadening our understanding of how the hippocampus represents information. By demonstrating its involvement in multimodal integration, the study provides new insights into the broad role the hippocampus plays in cognition and how it supports our rich, multimodal everyday experiences. The findings underscore the complexity of neural representations and the need for sophisticated analytical methods to fully capture them.

This article is a Primer, a format that provides a concise introduction into an important aspect of biology highlighted by a current PLOS Biology research article. The author, Halle R. Dimsdale-Zucker, is affiliated with the Department of Psychology at the University of California, Riverside. The Primer was published on November 4, 2026, and is distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. The authors received no specific funding for this work and have declared that no competing interests exist.

The study by Raccah and colleagues, which is the subject of this Primer, was designed to address a specific disconnect in cognitive neuroscience. Much of what is known about how the brain represents memories comes from studies of impoverished stimuli, such as words or pictures presented in isolation. These controlled laboratory studies often strip down a memory to its component parts, which does not capture the complexity of everyday experiences. A long history of work has suggested a role for the hippocampus, a key memory-related brain structure tucked inside the temporal lobes, in representing and remembering visual information. This includes pictures of objects and complex scenes, as well as linking information, or binding, experiences to their surrounding contexts. The ability to bind or link together experiences is one of the processes that helps to support autobiographical memories for rich everyday experiences, such as having coffee with a friend.

Memory-related representations in the hippocampus have been shown to represent specific memories, such as when you had coffee with a friend at a specific location. They also represent commonalities across similar events, such as all the different times you have hung out with that friend or similarities amongst all of the times you went to that specific coffee shop, or even across visits to many different coffee shops. However, the previous focus on visual stimuli left a gap in understanding how the hippocampus handles the multimodal nature of real-world events, where sights, sounds, and other sensory inputs are combined.

The new study attempted to close this gap by using stimuli that more closely approximate everyday experiences, such as television shows and movies. While previous work using such stimuli has helped to bridge the gap between controlled laboratory work and complex real-world experiences, it did not fully explain which features of multimodal stimuli are represented by the hippocampus. The current research sought to disentangle hippocampal representations for the same complex stimuli by selectively decomposing visual-only or auditory-only information. It also examined how the combination of these features changes these representations when sensory streams are congruent or incongruent.

The findings of the study highlight the importance of using different analytical approaches to understand the hippocampus. By looking across activity in an entire region and by looking across distributed activity profiles, the researchers were able to reveal nuanced differences that might be missed by a single method. The univariate analyses provided a broad view of overall activation, while the multivariate analyses offered a more detailed perspective on how specific patterns of neural activity corresponded to different stimulus types. This dual approach is essential for capturing the complexity of neural representations in the hippocampus, which is a structure with a broad role in cognitive operations ranging from attention and language to perception and prediction.

The study’s focus on the hippocampus’ role in representing multimodal stimuli contributes to a broader understanding of how this brain structure supports our rich, multimodal everyday experiences.

Subject of Research: Hippocampal representation of unimodal and multimodal audiovisual stimuli

Article Title: Understanding the role of the hippocampus in representing multimodal stimuli

Article References: Dimsdale-Zucker, H. R. (2026). Understanding the role of the hippocampus in representing multimodal stimuli. PLOS Biology, 24(11), e3004058. https://doi.org/10.1371/journal.pbio.3004058

Image Credits: AI Generated

DOI: 10.1371/journal.pbio.3004058

Keywords: hippocampus, multimodal stimuli, fMRI, memory, cognitive neuroscience, audiovisual integration, PLOS Biology, neural representation, Understanding, role, representing, multimodal