"What we have done is quantify it, automate it, prove its effectiveness and teach it very effectively."

When trying to lie your way through any situation, keep a tight rein on your zygo maticus major and your orbicularis oculi. They'll give you away faster than a snitch.

So says social psychologist Mark Frank, whose revolutionary research on human facial expressions in situations of high stakes deception debunks myths that have permeated police and security training for decades. His work has come to be recognized by security officials in the U.S. and abroad as very useful tool in the identification and interrogation of terrorism suspects.

By applying computer technology to the emotion-driven nature of nonverbal communication, Frank, a professor of communication in the School of Informatics at the University at Buffalo, has devised methods to recognize and accurately read the conscious and unconscious behavioral cues that suggest deceit.

His research already is employed by investigative bodies around the world and, Frank says, "It can be applied to the training of security checkpoint personnel to help them identify and decode 'hot spots,' the subtle conversational cues and fleeting flashes of expression that betray buried emotions or suggest lines of additional inquiry."

Frank notes that a large body of prior research has elaborated and sharpened Darwin's observations about the evolutionarily-derived nature of emotion and its expression.

In fact, Frank's mentor during his post-doctoral years at the University of California, San Francisco, was Paul Ekman, the world's foremost expert in reading facial expressions. Ekman conducted extensive cross-cultural research and found that a wide range of facial expressions related to specific emotions are identical from culture to culture.

He found that subjects' tics, furrows, smirks, frowns, smiles and wrinkles as they emerge in assorted combinations offer surprisingly accurate windows to the emotions.

"Fleeting facial expressions are expressed by minute and unconscious movements of facial muscles like the frontalis, corregator and risorius," Frank says, "and these micro-movements, when provoked by underlying emotions, are almost impossible for us to control."

Ekman and his colleague Wallace Friesen came up with a numbering system for all of these movements: for example, left and right eyebrows up is 1; down, 2; eyebrows pulled together, 4; upper eyelid raised, 5, and so on and related them to expressions of various emotion that are found the world over.

Building on their research, Frank has identified and isolated specific and sometimes involuntary movements of the 44 human facial muscles linked to fear, distrust, distress and other emotions related to deception.

Then, in a project for the National Science Foundation, he developed computer programs that automated Ekman's numbering process, making it possible to identify automatically every facial expression, including those tied to deceit, shown by subjects in taped interviews. Before this automation was developed, it took up to three hours of playing, rewinding and replaying, videotapes to analyze a single minute of blinks and twitches.

Frank's system has proven successful in identifying suspects involved in conventional criminal and potentially criminal behavior. It is now being tested for use in identifying potential terrorists.

"I want to make it clear that one micro-expression or collection of them is not proof of anything," Frank says. "They have meaning only in the context of other behavioral cues, and even then are not an indictment of an individual, just very good clues."

J.J. Newberry, formerly of the federal Bureau of Alcohol, Tobacco and Firearms, calls Frank and his methods "uncanny."

They are so effective that although he does not advertise his work nor actively solicit contracts in the field, Frank been asked to assist judges; health and police agencies, including the Los Angeles Police Department, the U.S. Federal Judiciary, the Bureau of Alcohol, Tobacco and Firearms and Department of Homeland Security, and other legal, medical and law enforcement communities from Singapore to Scotland Yard.

Since 9/11, a variety of federal government agencies have provided funding for Frank, although he declines to discuss the precise nature of his current research until it is complete and published.

In the course of his work with various investigative units, Frank says that, in addition to teaching them how to recognize behavioral cues, he has successfully advocated the use of a "rapport building" style of communication in interviews, because it is much more effective than the hostile/accusatory styles used in the past.

Frank says he began to develop identification skills when he was bouncer in a Buffalo bar. He says he trained himself to spot behavior that suggested that patrons were underage, packing a .22 or itching for a fight. He developed a sixth sense that allowed him to spot potential troublemakers by the way they looked when they walked in - "like they were trying to get away with something," he says. These were, for the individuals in question, high-stakes situations.

He honed his skills during years of research by staring at miles of videotape (sometimes in slow motion) in which crooks, sneaks and killers proclaimed their innocence, or hundreds of volunteer student liars tried to earn a little cash by successfully deceiving their interviewers.

"This identification skill is one that some police employ successfully. They work in a high-stakes profession that helps them develop what they would call an acute intuitive sense," says Frank, the son of a Buffalo police officer

"What we have done is quantify it, automate it, prove its effectiveness and teach it very effectively."

Emotions

Introduction
Emotion is one of the most controversial topics in psychology, a source of intense discussion and disagreement from the earliest philosophers and other thinkers to the present day. Most psychologists can probably agree on a description of emotion, e.g., what phenomena to include in a discussion of emotion. The enumeration of these parts of emotion are called the "components of emotion" here. These components are distinguished on the basis of physiological or psychological factors and include emotion faces, emotion elicitors, and emotion neural processes.

Components of Emotion
Common representation of angry emotion experience:"steamed up" with hot glowing eyes, and uncontrolled appearance. Is it the same across people?
Interpersonal aggression in the form of instrumental behaviors produced by skeletal muscles is often a concomitant of anger.

A bright idea can bring a pleasant emotion, or pleasant emotions can foster bright ideas.

The component that seems to be the core of common sense approaches to emotion, the one that most people have in mind when talking about human emotions, is the feeling component, i.e., the passion or sensation of emotion. For example, people generally agree that the state of mind during anger is different from that when one is happy.

This component is also one of the most contentious in scientific discussions of emotion, raising many questions such as:

to what extent are such feelings, especially the claimed differences in quality, based on real physical differences?
is the feeling quality of a particular emotion shared among people?
what is the nature of the differences in quality among emotions?
what underlies or produces these feelings?
what importance or function do such feelings have?

Another obvious descriptive component of emotion is the set of behaviors that may be performed and observed in conjunction with an emotion. These behaviors are produced by the striated muscular system and are of two general types: gross behaviors of the body effected by the skeletal muscles and the so-called emotion expressions. These categories shade into each other because any behavior can be interpreted as expressing emotion. The gross body behaviors may have no apparent adaptive value, e.g., wringing and rubbing the hands or tapping a foot, or they may be directed towards a goal, e.g., striking something or running away. In the field of animal behavior, discovering the adaptive function and organization of behaviors in situations analogous to human emotion, and speculating on the evolutionary patterns of these behaviors is an established endeavor. This emphasis has not typically been given to the study of human emotions by psychologists. The facial and bodily behaviors called "emotion expressions" are indicators of emotion, as opposed to effecting some action or achieving some goal. These expressions can differentiate one emotion from another. The most widely discussed and investigated emotion expressions are the emotion faces (see the examples of emotional expressions).

Adrenalin is a secretion that affects many organs and may contribute to the felt quality of emotion. A less obvious component of emotion is the set of internal bodily changes caused by the smooth muscles and glands. Chemicals secreted by the body's various glands are activated during emotion and spread to other parts of the body, usually by the blood, to act in diverse ways on the nervous system and other organs. Smooth muscles of the digestive system, circulatory system, and other bodily components can shift from their typical level or type of operation during emotion under the effects of chemical and neural action. This component includes some behaviors that can be observed, such as the constriction or dilation of the iris of the eye, possibly piloerection, and sweating, blanching, and flushing of the skin, and other responses that are relatively hidden, such as heart rate, stomach activity, and saliva production.

Computers often elicit frustration and anger. Another less observable component in emotion consists of the ideation, imagery, and thoughts that occur during emotion. These aspects of emotion are also cognitive activities, and can both give rise to an emotional event and be affected by it, e.g., thinking about a lost pet may evoke feelings of sadness, which may in turn evoke memories of a romance now finished. Since thoughts and other cognitions, like feelings, cannot be directly observed and are hard to measure, there is less understanding of how they fit into the emotion picture than other components.

The circumstances that give rise to emotions comprise another component, called the "elicitors" of emotion. These elicitors might be internal or external to the organism, e.g., a frightening pain in one's chest or a frightening dog at one's heels. Some events seem to activate similar emotion in people of all cultures, for example, the death of one's own child typically elicits sadness. Other things, such as what foods are relished or rejected with disgust, vary widely according to acculturation.

Finally, the neural processes that underlie much of the preceding activities can be considered a component of the emotion process, especially how the neurons and their emotional concomitants are organized centrally in the brain. Many contemporary research studies, and thus a lot of the research money, is focussed on anatomical and functional aspects of brain activity in regard to emotion.

Theories of Emotion
Beyond the descriptive approach to emotion, there are theories of emotion, which attempt to specify the interrelationships among components as described above and the causes, sources, and functions of emotional responses. Disagreement characterizes the intellectual climate surrounding emotion theories, but there are several works in print that summarize these approaches for the interested reader. The Theories of Emotion page of this section summarizes some of the most important theoretical statements on emotion that emphasize the role of the face.

Expression of Emotion
Emotion expression is another area of controversy, but at the descriptive level, some behaviors tend to occur with other components of emotion, and seem to reveal the quality of the emotion to an observer. The Emotion Expressions page of this section discusses the relations between emotion and facial expression.

Facets

Introduction to Facets - Special Topics Involving Facial Appearance and Behavior

The human face is related to a number of special interest topics, involving facial appearance, physiology, and behaviors. This site refers to these topics as facets, and discusses them on separate pages for convenient access. Facets are often in the news, of topical concern or interest, or matters for enduring discussion. The current facets are: Facial Aging, Facial Attractiveness, Facial Deception, and The Face in Health and Disease. Facial Aging discusses how the face ages, how facial identity is preserved over the life span, and how to combat the effects of aging, such as wrinkles, with cosmetics or more drastic actions. Facial Attractiveness discusses what makes a face beautiful, how attractiveness affects other people's attitudes, and how to maintain facial attractiveness. The Deception page points out the context of facial deception and discusses how to detect deception. The Health & Disease page shows how facial appearances reflect certain disease processes and how the face can actually predict what diseases a person may be particularly vulnerable to developing. Additional facets are in the works, so check back later for updates.

Anatomy of the Human Skull and the Nature of Physiognomy

Anatomy of the Human Skull
The human cranium and the facial bones are the foundation for the soft tissues of the face and head. Thus, much of the visible appearance of the human face depends upon the shapes and qualities of these bones. The cranium is that part of the skull that holds and protects the brain in a large cavity, called the cranial vault. Eight plate-like bones form the human cranium by fitting together at joints called sutures. The most important of these cranial bones for the appearance of the face is the frontal bone, which underlies the top of the face above the eyeballs. The human skull also includes 14 facial bones that form the lower front of the skull and provide the framework for most of the face that is important to psychological research. These 22 skull bones form other, smaller cavities besides the cranial vault, including those for the eyes, the internal ear, the nose, and the mouth. The important facial bones include the jaw bone or mandible, the maxilla or upper jaw, the zygomatic or cheek bone, and the nasal bone.

The shapes and features of the human skull determine much of the static appearances of the face and provide the basis for the features of physiognomy. Forensic pathologists and biologists can reconstruct the superficial appearance of a face merely from the human skull, as in the case of the Kennewick Man. The reconstruction of this skull revealed a facial appearance that indicates he is a descendant of a more ancient migration from Asia than that which brought the ancestors of the Indians (Amerinds), who settled widely in the Americas before the arrival of Europeans.

The skull bones are associated with many other features. Processes are areas where the bones have extra tissue to hold muscles and ligaments; lines are grooves in the bone from other developmental processes; foramina are holes in the bones through which nerves and blood vessels pass; sinuses are empty spaces in the bones that make the skull lighter. Some of these features affect the physiognomy of the face due to variations in thickness, size, location, and shape.

The diagrams above show the major external features of the human cranium and the major skull bones. The names in black are facial bones, those in red are cranial bones, and those in blue are features of the bones.

The Nature of Physiognomy
The term "physiognomy" refers to features of the face, especially so when, in the narrow sense, these features are used to infer the relatively enduring character or temperament of an individual. Physiognomy connotes a broader meaning, i.e., it refers to relatively unchanging facial features that might convey messages about any inner or hidden aspect of the person. Most of these facial features have as their basis the bony structure of the skull, on which the soft tissues lie. These features include the shapes and positions of major areas and landmarks of the face, such as the forehead, eyebrows, nose, cheeks, and mouth. The important facial features can be fairly accurately reconstructed by experts from the skull alone.

A diagram of the human cranium shows the major features of the skull, from which much of the visible appearances of the face can be extrapolated. Other physiognomic features are not directly linked to the bony skull, such as skin texture and coloration, hair placement and texture, and detailed shapes of fleshy features. All of these features change slowly and relatively little over time, and they are the sign vehicles for physiognomic messages. Proposing an association between these facial features and other aspects of the person, including personality, character, outcomes of medical treatment, romantic compatibility,or the destiny of the person, is a physiognomic approach. The validity of the association or inference based on physiognomy is a separate issue that can be established or discredited by empirical evidence. Accurately face reading these signs depends upon knowing which relations are valid and which are spurious.

Types of Messages
What might such physiognomic messages be about? Well, logically, as the signs of physiognomy change little or slowly, they can only be about characteristics that are relatively enduring and that change little or not at all. Such messages might include a person's genetic background (e.g., race, ethnicity, and family membership), genetic diseases (e.g., Down's syndrome), and more fuzzy concepts such as personality, character, and temperament. These facial features are unlikely to convey messages about characteristics that change rapidly or often, such as a transient emotions, because they cannot capture such rapid changes in their time scale of change. However, the possibility remains that repeated transient experiences, such as an often elicited emotion, might cumulate an effect on such slowly changing features. Also, observers often confuse some of these permanent signs with transient signs that actually do convey information about rapidly changing characteristics.

Topics related to physiognomy have a very long history in human cultures. In China and other Asian cultures, formal systems of face reading techniques developed sometime in the first millennia b.c.e., integrated with religious beliefs such as Confucianism. Substantial confidence in such methods developed in these cultures, and physiognomic inferences included descriptions of character, suitability for certain positions, and predictions about life and death. In Western cultures, the association of facial features with a person's characteristics also has a history, first noted in the writings of the ancient Greeks. Much later, several pseudo-scientific and cultish movements exploited the inference of character from physiognomic features. The physiognomy movement proper (which cultivated the narrow connotation for this term) was Phrenology, popularized by the 18th century Swiss philosopher Lavater. Some other applications of physiognomy are discussed further on the Physiognomy Applications page.

The face, despite recent advances in assessing identity such as biometrics and DNA testing, remains paramount in ordinary experience for identifying a individual person. The relatively permanent features of the face convey most of the information about identity, although styles in the production of more transient signals and other body shapes and sizes may also contribute to identity information. The signs of identity can be preserved in representations as schematic as the monochromatic drawing at the right, which Americans can easily identify as George Washington. Such permanent features of the face also convey information about the genetic background of the individual, including ancestry and ethnicity.

Tools for Studying Physiognomy
Visage is a project that attempts to represent the features that are used to describe the face and the characteristics that are associated with such facial features. A relational database stores the feature names, the characteristics, and the relations among them. You can get an idea of what this database contains from the Visage applet. It shows illustrations of a limited set of facial features that you can use to describe a face, then retrieves some of the descriptions that have been associated with these features.

Facial Expression

Facial Expression: A Primary Communication System

Expression implies a revelation about the characteristics of a person, a message about something internal to the expresser. In the context of the face and nonverbal communication, expression usually implies a change of a visual pattern over time, but as a static painting can express a mood or capture a sentiment, so too the face can express relatively static characteristics (sometimes called physiognomy).

The concept of facial expression, thus, includes:
1. a characteristic of a person that is represented, i.e., the signified;
2. a visual configuration that represents this characteristic, i.e., the signifier;
3. the physical basis of this appearance, or sign vehicle, e.g., the skin, muscle movements, fat, wrinkles, lines, blemishes, etc.; and
4. typically, some person or other perceiver that perceives and interprets the signs.

The existence and relationships among these components is a large area for study in the psychological and behavioral sciences. To read more about nonverbal communication, facial signs, and sign vehicles, see the on-line document Essential Behavioral Science of the Face ....

Facial expressions are an important channel of nonverbal communication. Many animal species display facial expressions, but expressions are highly developed particularly in the primates, and perhaps most of all, in humans. Even though the human species has acquired the powerful capabilities of a verbal language, the role of facial expressions in person-to-person interactions remains substantial. Messages of the face that provide commentary and illustration about verbal communications are significant in themselves. To see examples of such nonverbal communication, see the FAQ.

Other types of expressions provide another, different mode for understanding the private, hidden side of the inner person, a side which may not be accessible in the form of verbalizations. For example, the facial behaviors related to emotion can reveal part of the feeling side of a person's private life. Such emotion indicators range from stereotyped, full-face expressions that are obvious to fleeting, partial-face movements that are hard to see. For more information about emotion expressions and some pictures of facial expressions, see the Emotion section.

The study of human facial expressions has many aspects, from computer simulation and analysis to understanding its role in art, nonverbal communication, and the emotional process. Many questions about facial expressions remain unanswered and some areas are relatively unexplored. To get a broad picture of the kinds of questions that have been asked, answers to some of these questions, and the scientific research about the face that needs to be completed to answer them, see the online document Understanding the Face: Report to the National Science Foundation. Facial expressions and the ability to understand them are important for successful interpersonal relations, so improving these skills is often sought. See the Guide: How to Read Face for tips on improving your abilities.

Facial muscles produce facial expressions and emotion faces.
Tools for Studying Facial Expression Produced by Muscular Action
The Facial Action Coding System (FACS) is a method that researchers use to measure facial expressions by identifying the muscular activity underlying transient changes in facial appearance. Researchers use in facial analysis to determine the elementary behaviors that pictures of facial expressions portray. The FACS Affect Interpretation Database (FACSAID) is a tool for understanding what the muscular actions that FACS measures mean in terms of psychological concepts. FACSAID interprets the facial expressions in terms of meaningful scientific concepts.

Facial Action Coding System

The Facial Action Coding System (FACS) Manual is a detailed, technical guide that explains how to categorize facial behaviors based on the muscles that produce them, i.e., how muscular action is related to facial appearances. It illustrates appearance changes of the face using written descriptions, still images, and digital video examples. Behavioral scientists, CG animators, computer scientists interested in pattern recognition programs, and other technicians and scientists use FACS in their professional work when they need to know the exact movements that the face can perform, and what muscles produce them. Working through the exercises of the FACS Manual may also enable greater awareness of and sensitivity to subtle facial behaviors that could be useful for psychotherapists, interviewers, and other practitioners who must penetrate deeply into interpersonal communications.

FACS is a training manual, not necessarily easy reading, with lessons for detecting, performing, and categorizing facial movements. The manual does not discuss what the facial appearances described mean, except briefly in the Investigator's Guide. The FACS Manual enables the practitioner to recognize the elements of facial behavior that combine to create meaningful communications; FACS teaches the "alphabet" but leaves the considerable issue of semantics to other works. The FACS Investigator's Guide explains in general how to use FACS in scientific research, how it compares to other facial measurements, and what its psychometric properties are. The new version (2002) of FACS by Paul Ekman, Wallace V. Friesen, and Joseph C. Hager is now available for purchase. You can read about the new version of the FACS Manual, and purchase it on this site.

What is Facial Action Coding System?

Facial Action Coding System (FACS) is the most widely used and versatile method for measuring and describing facial behaviors. Paul Ekman and W.V. Friesen developed the original FACS in the 1970s by determining how the contraction of each facial muscle (singly and in combination with other muscles) changes the appearance of the face. They examined videotapes of facial behavior to identify the specific changes that occurred with muscular contractions and how best to differentiate one from another. They associated the appearance changes with the action of muscles that produced them by studying anatomy, reproducing the appearances, and palpating their faces. Their goal was to create a reliable means for skilled human scorers to determine the category or categories in which to fit each facial behavior. The FACS Manual was first published in a loose-leaf version with video or film supplements in 1978.

FACS measurement units are Action Units (AUs), not muscles, for two reasons. First, for a few appearances, more than one muscle was combined into a single AU because the changes in appearance they produced could not be distinguished. Second, the appearance changes produced by one muscle were sometimes separated into two or more AUs to represent relatively independent actions of different parts of the muscle. (After all, facial muscles were identified and named by anatomists, not behavioral psychologists.)

A FACS coder "dissects" an observed expression, decomposing it into the specific AUs that produced the movement. The scores for a facial expression consist of the list of AUs that produced it. Duration, intensity, and asymmetry can also be recorded. FACS scores are descriptive only, and provide no implications about the meaning of the behavior. Analysis of the data can use only these raw FACS scores, or the scores can be translated into more psychologically meaningful concepts by techniques such as FACSAID, a database interpretation system available to researchers.

Think you are good at reading people?

Think you're good at reading people? Most people feel they are, but actually fail miserably at it, confusing a half smile with approval when it signals contempt, or accepting an expression of apparent confidence while missing the concealed fear that lies beneath it.

Misreading facial expressions and the emotions underlying them results in a lot of misunderstandings and miscommunication. Often the failure comes from an inability to recognize minute expressions -- micro-expressions that flash across a face for less than a 15th of a second -- that reveal the true emotions a person may be uncomfortable expressing or is simply trying to conceal.

"These expressions tend to be very extreme and very fast," said Paul Ekman, professor of psychology at the University of California, San Francisco, School of Medicine and an expert in the physiology of emotion and nonverbal communication. "Eighty to 90 percent of people we tested don't see them."

Micro-expressions represent "the most extreme expressions human beings can make in a very fast period of time" and usually involve the entire face. Subtle expressions are easily overlooked because they involve minor movement in parts of the face -- raised eyelids that might signal the beginnings of fear or surprise, or the angled upturn of the inner eyebrows that might signal the beginnings of sadness.

But a new set of CD-ROMs developed by Ekman can help people recognize emotional "leakage" -- facial expressions that signal when a person is willfully suppressing or unconsciously repressing an emotion.

Training anyone, in under an hour, to spot fleeting expressions and interpret emotions they might otherwise miss if they were distracted by a person's gestures or tone of voice.

The Micro emotions are concealed emotions, and the Subtle Expression which explores more subtle expressions that occur when someone is just beginning to feel an emotion.

There are seven emotions universally expressed by all cultures -- anger, fear, disgust, surprise, sadness, happiness and contempt (interpreted as moral superiority).

Ekman said people most often confuse the expressions for fear and surprise, as well as the ones for anger and disgust, because they involve some of the same muscles.

Ekman's daughter, Eve, is the model in the SETT CD, which runs through the seven emotions as they might flit across various parts of the face -- for instance, the wrinkled nose that universally signifies disgust (or perhaps simply the smell of bad fish).