How sounds, shapes, speech and body movements convey emotion through one shared property

Death metal band logos often have a spiky look while romance novel titles often have a swirly script. The jaggedness or curviness of a font can be used to express an emotional tone. A Dartmouth study published in the Proceedings of the Royal Society B finds that sounds, shapes, speech and body movements convey emotional arousal the same way across the senses. The findings explain why nearly anything can have an emotional tone, including art, architecture and music.

“Our study set out to better understand how we express and read emotional arousal, which is fundamental to our core emotional state. We wanted to see if there is a low-level mechanism that allows us to decode emotional arousal information from the movements and sounds that people and animals make,” says lead author Beau Sievers, a postdoctoral student of psychology at Harvard University, who was a graduate student in the department of psychological and brain sciences at Dartmouth at the time of the study. “Our results show how the spectral centroid, or the balance of high-frequency versus low-frequency energy present in sounds, shapes and movements, allows us to express and understand emotional arousal,” adds Sievers.

The spectral centroid is essentially a multi-sensory measure of spikyness. The results explain why Zen gardens and brutalist architecture have very different emotional effects, as well as why things like clouds and lullabies seem to go together even though one is seen and the other is heard: We match them based on the spectral centroid.

“In a series of studies, we demonstrate that people automatically perceive the frequency spectrum of whatever is coming into their ears and eyes and compute the average–the spectral centroid,” explains senior author Thalia Wheatley, a professor of psychological and brain sciences at Dartmouth, and principal investigator of the Dartmouth Social Systems Laboratory . “This is how people quickly identify the amount of emotional arousal in a person’s voices and movements but also in abstract shapes and sounds, such as why spiky shapes seem to convey higher arousal than rounded shapes,” she added.

To test whether the spectral centroid is used to express and understand emotional arousal, the researchers conducted five mini-studies, some of which asked participants to make judgements about the emotional arousal of shapes, sounds and movements. The researchers tested if the spectral centroid of the stimulus could be used to predict participants’ emotional arousal judgements. The following are highlights from three of the mini-studies:

• The authors used a computer program to randomly create hundreds of shapes and sounds. Participants were asked to look at shapes and listen to sounds and judge their levels of emotional arousal. As the study reports, shapes and sounds which had a high spectral centroid were associated with high-arousal emotions (angry, excited), whereas the lower spectral centroid shapes and sounds were associated with low-arousal emotions (sad, peaceful).

• Participants were asked to draw shapes that were angry, sad, excited, or peaceful. The researchers then estimated the spectral centroids of the drawings by counting how many corners they had. The results revealed that angry and excited shapes had between 17 and 24 corners on average, while sad and peaceful shapes had between 7 and 9 corners on average. The spectral centroid could be used to predict the emotional arousal of shapes with close to 80 percent accuracy.

• The researchers examined real-world recordings of people’s body movements or of people speaking (in German), to see if the spectral centroid of the voices and movements could be used to predict participants judgements of emotional arousal. The researchers found that higher spectral centroids predicted judgements of higher emotional arousal.

The researchers explain that multi-sensory associations with emotions have been known for a long time but why they occur has been a mystery until now.

###

Sievers and Wheatley are available for comment at: [email protected] and [email protected]. Caitlyn Lee in the department of psychological and brain sciences at Dartmouth and William Haslett in the department of biomedical data science at the Geisel School of Medicine at Dartmouth also served as co-authors of the study.