Should You Trust Facial Expressions?

Written By: Angela Hu

A smile means happiness. A scowl indicates annoyance. A frown means sadness, and raised eyebrows mean confusion – or does it? Can we trust facial expressions to tell us how someone feels?

The short answer is: not really. In studies, researchers have found that attempts to figure out emotions based on a person’s facial expressions are almost always wrong. A review of more than 1000 studies found that emotions are expressed in such a variety of ways that you can’t rely on simply their facial expression. 

People may express their emotions differently for a number of reasons, such as their culture. In some cultures, it isn’t as common to smile as much in public. In addition, even if you are happy it doesn’t necessarily guarantee a smile. Similarly, people on average scowl less than 30% of the time when angry. Not scowling does not mean no anger, and scowling might mean other emotions such as concentration.

Many other studies that say otherwise may have flaws. These include hiring actors that pull exaggerated faces or giving participants a limited number of labels to choose from for each facial expression.

Why Does This Matter?

Companies are developing AI technology that connects facial movements to emotion or intent. This can tell the company whether a customer is satisfied, or whether a person should be hired. Some companies require a video in the application, which is then analyzed by AI. If a candidate appears bored or uninterested, they may be less likely to be hired.

These claims can be dangerous. It is missing a person’s real emotion and ignores other factors. These can include body language, head movements, or even facial colour such as blushing. In an experiment, participants were shown a man’s face open in a “scream” with a bright red face. Most people thought he was angry, but he was actually an athlete celebrating a goal. 

I think it’s extremely important that we recognize that facial expressions will never be enough to recognize intent. Moving forward, programmers should continue to see where technology might not be enough.