Automatic Emotion Recognition Using Facial Expression: A Review

International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395 -0056
Volume: 03 Issue: 02 | Feb-2016 www.irjet.net p-ISSN: 2395-0072
© 2016, IRJET | Impact Factor value: 4.45 | ISO 9001:2008 Certified Journal | Page 488
Automatic Emotion Recognition Using Facial Expression: A Review
Monika Dubey1, Prof. Lokesh Singh2
1Department of Computer Science & Engineering, Technocrats Institute of Technology, Bhopal, India
2Asst.Professor, Department of Computer Science & Engineering, Technocrats Institute of Technology,Bhopal, India
---------------------------------------------------------------------***---------------------------------------------------------------------
Abstract - This paper objective is to introduce needs and
applications of facial expression recognition. Between Verbal
& Non-Verbal form of communicationfacialexpression isform
of non-verbal communication but it plays pivotal role. It
express human perspective or filling & his or her mental
situation. A big research has been addressed to enhance
Human Computer Interaction (HCI) over two decades. This
paper includes introduction of facial emotion recognition
system, Application, comparative study of popular face
expression recognitiontechniques &phasesofautomaticfacial
expression recognition system.
Key Words: Emotion recognition, Facial expression,
Image processing, Human Machine Interface.
1. INTRODUCTION
Emotional aspects have huge impact on Social intelligence
like communication understanding, decision making and
also helps in understanding behavioral aspect of human.
Emotion play pivotal role during communication. Emotion
recognition is carried out in diverse way, it may be verbal or
non-verbal .Voice (Audible) isverbal formofcommunication
& Facial expression, action, body postures and gesture is
non-verbal form of communication. [1] While
communicating only 7% effect of message is contributes by
verbal part as a whole, 38% by vocal part and 55% effect of
the speaker’s message is contributed by facial expression.
For that reason automated & real time facial expression
would play important role in human and machine
interaction. Facial expression recognition would be useful
from human facilities to clinical practices. Analysis of facial
expression plays fundamental roles for applications which
are based on emotion recognition like Human Computer
Interaction (HCI), Social Robot, Animation, Alert System &
Pain monitoring for patients.
This paper presents brief introduction of facial
expression in section I. Section II describes six universal
facial expressions and features. Section III gives brief detail
on comparative study of popular techniques proposed
earlier for Automatic Facial Emotion Recognition System.
Section IV includes phases of Automatic Facial Emotion
Recognition System. Section VincludesApplicationsofFacial
Emotion Recognition System.
2. CATEGORIZING FACIAL EXPRESSIONS & IT’S
FEATURES:
Facialexpressionpresentskeymechanismtodescribehuman
emotion. From starting to end of the day human changes
plenty of emotions, it may be because of their mental or
physical circumstances. Although humans are filled with
various emotions, modern psychology defines sixbasicfacial
expressions: Happiness, Sadness, Surprise, Fear, Disgust,
and Anger as universal emotions [2]. Facial muscles
movements help to identify human emotions. Basic facial
features are eyebrow, mouth, nose & eyes.
Table -1: Universal Emotion Identification
Universal Emotion Identification
Emotion Definition Motion of facial part
Anger
Anger is one of the most dangerous
emotions. This emotion may be harmful
so, humans are trying to avoid this
emotion. Secondary emotions of anger are
irritation, annoyance, frustration, hate
and dislike.
Eyebrows pulled down,
Open eye, teeth shut and
lips tightened, upper and
lower lids pulled up.
Fear
Fear is the emotion of danger. It may be
because of danger of physical or
psychological harm. Secondary emotions
of fear are Horror, nervousness, panic,
worry and dread.
Outer eyebrow down,
inner eyebrow up,
mouth open, jaw
dropped
Happiness
Happiness is most desired expression by
human. Secondary emotions are
cheerfulness, pride, relief, hope, pleasure,
and thrill.
Open Eyes, mouth edge
up, open mouth, lip
corner pulled up, cheeks
raised, and wrinkles
around eyes.
Sadness
Sadness is opposite emotion of
Happiness. Secondary emotions are
suffering, hurt, despair, pitty and
hopelessness.
Outer eyebrow down,
inner corner of
eyebrows raised, mouth
edge down, closed eye,
lip corner pulled down.
Surprise
This emotion comes when unexpected
things happens. Secondary emotions of
surprise are amazement, astonishment.
Eyebrows up, open eye,
mouth open, jaw
dropped
Disgust
Disgust is a feeling of dislike. Human may
feel disgust from any taste, smell, sound
or tough.
Lip corner depressor,
nose wrinkle ,lower lip
depressor, Eyebrows
pulled down

Fig -1: Six basic Facial Expressions
3. BACKGROUND ANALYSIS:
Humancan recognize emotions withoutanysignificantdelay
and effort but recognition of facial expressionbymachineisa
big challenge. Some of the vital facial expression recognition
techniques are:
3.1 Statistical movement based:
This paper proposed noise and rotation invariant facial
expression recognition which is based of Statistical
movement that is Zernike moments [3]. Extracted feature
form Zernike moments are given as input to Navie Bayesian
classifier for emotion recognition.
Pros:
i) The rotation invariance is achieved with the help of Zernike
moments.
ii)Recognition time less than 2 seconds for frontal face image.
Cons:
i)Emotion recognition system got affected because of rotation
of facial images.
3.2 Auto-Illumination correction based:
In this paper, facial expressions are determined using
localization of points called Action Unit (AU’s) without
labelling them [4]. Face is recognized by using the skin &
Chrominance of the extracted image. By using mapping
technique extracted eyes and mouth are mapped together.
Skin and non-skin pixels are separated to separatefacefrom
the background by using Haar-Cascaded method.Thispaper
is based on multiple face image recognition.
Pros:
i) Single and multiple face detection system.
ii)Limitation of illumination is removed and automatically
corrected using colour consistency algorithm.
Cons:
i) 60% recognition rate achievedwhile detectingmultipleface
images. So, it is required to achieve more accuracy.
ii)This system suffers under very poor lighting system.
3.3 Identification-driven Emotion recognition
system for a Social Robot:
In order to provide personalized emotion recognition,
this paper includes identification step prior to emotion
classification [5]. For finding facial configuration hybrid
approach used which includes Active Space models and
Active appearance models. Face tracker is used for face
detection. Texture information consistsofassetofvectorsto
describe the face of 3D model.
Pros:
i) i)Identification of subject and prior knowledge about the
subject enhance the performance recognition in term of
quality and speed of classification.
ii) ii)82% recognition rate when facial image taken in a social
robot working environment that includes various lighting
conditions and different positions and orientations of
subject face.
Cons:
i) i)Required training before using it as application of social
robot emotion recognition system.
ii)Required appropriate templateinthetracking datatocover
the whole 5features space for emotion recognition.
3.4 E-learning based emotion recognition system:
This paper proposed E-learning based emotion
recognition system [6].SVM (Support Vector Machine)
classifier based Adoost algorithm used to locate humanface.
Ad boost algorithm compares the classifier by extracting
features with week classifier to strong classifier. This is
iterative weight updating process.
Pros:
i)This paper presents application for the emotion in network
teaching system.
ii)Wearing glasses on the face area has no effect on emotion
recognition.
Cons:
i) i) Distance between the camera and face will have an impact
on an area of face recognition.
ii)Regional impact of the humanfaceeffecttheperformanceof
emotion recognition like- Hear, Sitting postures, Light
strength.
3.5 Cognitive Face Analysis System for Interactive
TV System:
This paper proposed, emotion detection of members
watching TV Program [7]. Face expression recognition are
employed to identify specific TV viewer and recognize their
internal emotional state.Ada-LDAmethodbasedrecognition.
Per second over 15 frames can operated.
Pros:
i)This paper introduced a novel architecture of the future
interactive TV.
ii)Proposed technique is based on real time emotion
recognition system.
iii)It can operate at over 15 frames per second.

Cons:
i) Recognition rate differ with type of facial database used.
ii)Need to improve performance of recognition and timing for
real time application.
3.6 Motion detection based facial expression using
Optical flow:
In order to localize facial features approximately active
Infra-Red (IR) illumination used[8].SourceVector(SV)used
for vector collection which shows motion and deformation
due to emotion representation. Emotions are classified
according to the estimated similarity between the source
vector and execution motion vector and highest degree of
similarity could be identified as detected emotion.
Pros:
i) Few number of image frames (three frames)aresufficient to
detect facial expression.
ii)Not necessary to determine exact facial feature locations,
only the approximate values are sufficient.
Cons:
i) Recognition rate of emotion “Fear” is less than other
emotions.
Table -2: Comparative Study
Comparative Study
Title
Technique
Database Performance
(%)
Remarks
Statistical
Moments
based Facial
expression
Analysis
Feature
Extraction:
Zernike
moments
Classification:
Naive Bayesian
classifier
JAFFE
(Japanese
Female Facial
expression)
database 60
images used
for experiment.
Average
accuracy for
six emotions
is 81.66% in
time less than
2 seconds.
Emotion
accuracy
graph shows
highest
recognition
rate of
happiness
and lowest
recognition
rate of
sadness.
Facial
expression
recognition
with Auto-
Illumination
correction
Expressions on
the face are
determined
with Action
Units (AU’s)
Single and
Multiple face
image
60%
recognition
rate for
multiple face
image
Illumination
on image
plays vital
role.
Identification-
driven
Emotion
recognition
system for a
Social Robot
Hybrid
approach used
for
personalized
emotion
recognition,
MUG facial
expression
database used.
More than 50
people frontal
face database
used aged
between 20-25
years.
82%
performance
achieved with
KNN
Classifiers.
3D model
facial image
used.KNN
classifier
gives good
performance
for emotion
recognition.
The
application
study of
learner’s face
detection and
location in the
teaching
network
system based
on emotion
SVM(Support
Vector
Machine)
classifier based
Adaboost
algorithm used
PIE face image
database used
Detection and
Correction
rate 95% or
more.
Presents
application
of face
emotion
recognition
with of E-
learning
system.
recognition
Cognitive
Face Analysis
System for
Future
Interactive TV
Ada-LDA
learning
algorithm and
MspLBP
features used
for effective
multi-class
pattern
classifier
JAFFE and
MIT+CMU
database
Recognition
rate of over
15 frames per
second
Real time
performance
with high
recognition
rate
An Efficient
Algorithm for
Motion
Detection
Based Facial
Expression
Recognition
using Optical
Flow
Infra-Red(IR)
illumination
used for facial
feature
approximately
localization.
Source Vector
(SV) used for
vector
collection and
identification of
emotion is
based on
highest degree
of similarity
between source
vector and
execution
motion vector
Approximately
1000 images
sequences of
Cohn-Kanada
Facial
Expression
Database with
65% female
facial image
used for
experiment
94%
recognition
rate
Only three
frames are
sufficient to
detect facial
expression.
4.INTEGRATEDFACIALEXPRESSIONRECOGNITION
SYSTEM:
The system which performs recognition of facial expression
is called facial recognition system. Image processing is used
for Facial expression recognition. With the help of image
processing useful information from image can get extracted.
Image processing converts image into digital form and
perform some operations on it to extract useful information
from image.
Facial expression recognition system consists of
following steps:
4.1 Image Acquisition:
Static image or image sequences are used for facial
expression recognition.2-D gray scale facial image is most
popular for facial image recognition although color images
can convey more information about emotion such as
blushing. In future color images will prefer for the same
because of low cost availability of color image equipments.
For image acquisition Camera, Cell Phone or other digital
devices are used.
4.2 Pre-processing:
Pre-Processing plays a key role in overall process. Pre-
Processing stage enhances the quality of input image and
locates data of interest by removing noise and smoothing
the image. It removes redundancy from image without the
image detail. Pre-Processing also includes filtering and
normalization of image which produces uniform size and
rotated image.

4.3 Segmentation:
Segmentation separates image into meaningful reasons.
Segmentation of an image is a method of dividing the image
into homogenous, self consistent regions corresponding to
different objects in the image on the bases of texture, edge
and intensity.
4.4 Feature Extraction:
Feature extraction can be considered as “interest” part in
image. It includes information of shape, motion, color,
texture of facial image. It extracts the meaningful
information form image. As compared to original image
feature extraction significantly reduces the information of
image, which gives advantage in storage.
4.5 Classification:
Classification stage follows the output of feature extraction
stage. Classification stage identifies the facial image and
grouped them according to certain classes and help in their
proficient recognition. Classification is a complex process
because it may get affected by many factors. Classification
stage can also called feature selection stage, deals with
extracted information and group them according to certain
parameters.
Fig -2 Facial expression Recognition System
5. APPLICATION AREA:
With the rapid development of technologies it is required to
build an intelligent system that can understand human
emotion. Facial emotion recognition is an active area of
research with several fields of applications. Some of the
significant applications are:
i) Alert system for driving.
ii) Social Robot emotion recognition system.
iii) Medical Practices.
iv) Feedback system for e-learning.
v) The interactive TV applications enable the customer
to actively give feedback on TV Program.
vi) Mental state identification.
vii) Automatic counseling system.
viii) Face expression synthesis.
ix) Music as per mood.
x) In research related to psychology.
xi) In understanding human behavior.
xii) In interview.
6. CONCLUSION
Extensive efforts have been made over the past two decades
in academia, industry, and government to discover more
robust methods of assessing truthfulness, deception, and
credibility during human interactions. Efforts have been
made to catch human expressions of anyone. Emotions are
due to any activity in brain and it is known through face, as
face has maximum sense organs. Hencehumanfacial activity
is considered. The objective of this research paper is to give
brief introduction towards techniques, application and
challenges of automatic emotion recognition system.
REFERENCES
[1] A.Mehrabian, ”Communication without Words”
Psychology Today, Vol.2, no.4, pp 53- 56, 1968
[2] Ekman P, Friesen WV. Constants across cultures in the
face and emotion Journal of personality and social
psychology 1971; 17:124
[3] Bharati A.Dixit and Dr. A.N.Gaikwad ”Statistical
Moments Based Facial Expression Analysis” IEEE
International Advance Computing Conference (IACC),
2015
[4] S.Ashok Kumar and K.K.Thyaghrajan ”Facial
Expression Recognition with Auto-Illumination
Correction” International Conference on Green
Computing, Communication andConservationofEnergy
(ICGCE), 2013
[5] Mateusz Zarkowski “Identification-deiven Emotion
Recognition System for a Social Robot” IEEE, 2013
[6] Shuai Liu and Wansen Wang “The application study of
learner’s face detection and location in the teaching
network system based on emotion recognition” IEEE,
2010
[7] Kwang Ho An and Myung Jin Chung “Cognitive Face
Analysis System for Future Interactive TV”IEEE, 2009
[8] Ahmad R. Naghsh-Nilchi and MohammadRoshanzamir
“An Efficient Algorithm for Motion Detection Based
Facial Expression Recognition using Optical Flow”
International Scholarly and Scientific Research and
Innovation, 2008

BIOGRAPHIES
Monika Dubey received her B.E
degree in Computer Science from
Lakhmi Chand Institute of
Technology (LCIT), Bilaspur,
Chhattisgarh, 2012. Technocrats
Institute of Technology, Bhopal,
Madhya Pradesh, where she is
moving toward M.Tech degree in
Computer Science. Her current
interests includeimageprocessing,
Computer Vision.
Lokesh Singh received his B.E
MIT,Ujjain,Madhya Pradesh.M.E
Institute of Engineering and
Technology(IET), Indore, Madhya
Pradesh. Currently, he is working
as Asst. Professor in Technocrats
Institute of Technology, Bhopal,
Madhya Pradesh. His current
interests includeimageprocessing,
Computer Vision, Human
Computer Interaction.
nd
Author
Photo

Automatic Emotion Recognition Using Facial Expression: A Review

More Related Content

What's hot (20)

Similar to Automatic Emotion Recognition Using Facial Expression: A Review (20)

More from IRJET Journal (20)

Recently uploaded (20)

Automatic Emotion Recognition Using Facial Expression: A Review