Emotion-detection applications built on outdated science, report warns

Software that purportedly reads emotions in faces is being deployed or tested for a variety of purposes, including surveillance, hiring, clinical diagnosis, and market research. But a new scientific report finds that facial movements are an inexact gauge of a person’s feelings, behaviors or intentions.

“It is not possible to confidently infer happiness from a smile, anger from a scowl or sadness from a frown, as much of current technology tries to do when applying what are mistakenly believed to be the scientific facts,” a group of leading experts in psychological science, neuroscience and computer science write in their comprehensive research review.

The report appears in Psychological Science in the Public Interest, a journal of the Association for Psychological Science, and is authored by Lisa Feldman Barrett of Northeastern University, Ralph Adolphs of the California Institute of Technology, Stacy Marsella of Northeastern University and the University of Glasgow, Aleix M. Martinez of The Ohio State University and Seth D. Pollak of the University of Wisconsin-Madison.

The authors note that the general public and some scientists believe that there are unique facial expressions that reliably indicate six emotion categories: anger, sadness, happiness, disgust, fear, and surprise. But in reviewing more than 1,000 published findings about facial movements and emotions, they found that typical study designs don’t capture the real-life differences in the way people convey and interpret emotions on faces. A scowl or a smile can express more than one emotion depending on the situation, the individual or the culture, they say.

“People scowl when angry, on average, approximately 25 percent of the time, but they move their faces in other meaningful ways when angry,” Barrett explains. “They might cry, or smile, or widen their eyes and gasp. And they also scowl when not angry, such as when they are concentrating or when they have a stomach ache. Similarly, most smiles don’t imply that a person is happy, and most of the time people who are happy do something other than smile.”

In a separate article in the journal, Alan Cowen and Dacher Keltner of the University of California, Berkeley; Disa Sauter, University of Amsterdam; and Jessica L. Tracy of the University of British Columbia note that most scientists agree that facial expressions are meaningful, even if they don’t follow a one-to-one match with six basic emotion categories. They propose a new model for studying emotion-related responses in all their complexity and variations. This approach would measure not only facial cues, but also body movements, voice fluctuations, head movements and other indicators to capture such nuanced responses as smiles of embarrassment or sympathetic vocalizations, they say.

The report’s conclusions have broad implications, according to the authorship team. The FBI and the Transportation Security Administration have trained agents in the past to assess smiling, scowling and other facial movements to identify and stop potential terrorists. Law enforcement agencies in the United States and Europe are now experimenting with technologies designed to automate emotion detection through facial scans. Some companies are experimenting with software to track the facial movements of job applicants during interviews. Such technology might be able to detect facial movements, but they do not detect the psychological meaning of those facial movements, Barrett and co-authors say.

“We thought this was an especially important issue to address because of the way so-called ‘facial expressions’ are being used in industry, educational and medical settings, and in national security,” say Barrett and her co-authors.

###

The full report and commentary are available online at https://www.psychologicalscience.org/publications/emotional-expressions-reconsidered-challenges-to-inferring-emotion-from-human-facial-movements.html

For more information about this study, please contact: Lisa Feldman Barrett at [email protected] or 617-794-4409, Ralph Adolphs at [email protected], Stacy Marsella at [email protected], Aleix Martinez at [email protected], or Seth Pollak at [email protected] or 608-698-2523.

Psychological Science in the Public Interest is a journal of the Association for Psychological Science. It publishes an eclectic mix of thought-provoking articles on the latest important advances in psychology. For a copy of the article “Emotional Expressions Reconsidered: Challenges to Inferring Emotion from Human Facial Movements” and access to other research findings, please contact Anna Mikulak at 202-293-9300 or [email protected].