![International Journal of Trend in Scientific Research and Development (IJTSRD)
Volume 4 Issue 6, September-October 2020 Available Online: www.ijtsrd.com e-ISSN: 2456 – 6470
@ IJTSRD | Unique Paper ID – IJTSRD33267 | Volume – 4 | Issue – 6 | September-October 2020 Page 29
AI Therapist – Emotion Detection using Facial Detection and
Recognition & Showing Content According to Emotions
Sanket Godbole1, Jaivardhan Shelke2
1Student, 2Software Engineer,
1Departmet Information Technology, Government College of Engineering, Aurangabad, Maharashtra, India
2Kanthaka Consultancy Services Pvt Ltd, Pune, Maharashtra, India
ABSTRACT
This paper presents an integrated system for emotion detection using facial
detection and recognition. we have taken into account the fact that emotions
are most widely represented with eyes and mouth expressions. In this
research effort, we implement a general convolutional neural network (CNN)
building frameworkfor designing real-time CNNs.We validate our models by
creating a real-time vision system that accomplishes the tasks of face
detection, emotion classification, and generating the content according to the
emotion or mood of the person simultaneously in one blended step using our
proposed CNN architecture.
Our proposed model consisted of modules such as image processing, Feature
extraction, feature classification, and recommendation process. The images
used in the experiment are pre-processed with various image processing
methods like canny edge detection, histogram equalization,fitellipse,andFER
dataset ismediated for conducting the experiments. Withatrainedprofilethat
can be updated flexibly, a user can detect his/her behavior on a real-time
basis. It utilizes the state of the art of face detection and recognition
algorithms.
KEYWORDS: classification and recognition, Convolutional Neural Networks,
Feature Extraction
How to cite this paper: Sanket Godbole |
Jaivardhan Shelke "AI Therapist –
Emotion Detection using Facial Detection
and Recognition & Showing Content
According to
Emotions" Published
in International
Journal of Trend in
Scientific Research
and Development
(ijtsrd), ISSN: 2456-
6470, Volume-4 |
Issue-6, October 2020, pp.29-31, URL:
www.ijtsrd.com/papers/ijtsrd33267.pdf
Copyright © 2020 by author(s) and
International Journal of TrendinScientific
Research and Development Journal. This
is an Open Access article distributed
under the terms of
the Creative
Commons Attribution
License (CC BY 4.0)
(http://creativecommons.org/licenses/by
/4.0)
I. INTRODUCTION
Human emotions and its detection and recognition play a
very vital role in applications intended for handicapped
persons or robotic intelligence or persons with difficulty in
recognizing emotions or for developing intelligentGraphical
Application. To detect the facial expressions on the human
face his face must be detected and the face location and
position must be known. Hence in most cases, emotion
detection algorithms start with face detection, taking into
account the fact that face emotions are mostly depicted in
eyes and mouth.
There are two mutually benefited subtasks in any facial
detection and recognition task that include Detection and
Extraction of the image features. facial expression
recognition system detects a facewithin the captured image
and then it extracts features from the potential face for the
recognition task. There are many different typesofemotions
that influence how we liveand interactwith others.Attimes,
it may seem like we are ruled by these emotions. we intend
to find this type of emotion like sadness, anger, happiness,
disguise, etc.
Furthermore, CNNs are used as black-boxes and often their
learned features remain hidden, making it complicated to
establisha balance between their classificationaccuracyand
unnecessary parameters Therefore, we implemented a real-
time visualization.
II. Literature Review
IliasMaglogiannis, Demosthenes Vouyioukas, and Chris
Aggelopoulospresented [1] presented an emotion detection
and recognition system for color images. The proposed
system detects the skin color using MRFs estimation in
YCbCr space, overcoming the difficulty ofvarious skin
regions detection under different lighting conditions,
emotion recognition then follows, based on the construction
of face, eye, and mouth maps.
Octavio Arriaga and Matias Valdenegro [2] have proposed
and tested general building designs for creating real-time
CNNs. Their architecture has been systematically built to
reduce the number of parameters. by eliminating the fully
connected layers and by reducing the number of parameters
in the remaining convolutional layers via depth-wise
separable convolutions.
Bee Theng Lau [3] developed to help to detect the emotions
of the disabled who could not make a speech and also
physical movements easily. These peopletendtobeseatedat
one fixed location or wheelchair and require assistance in
most of the physical movements. the detection and
IJTSRD33267](https://image.slidesharecdn.com/6aitherapistemotiondetectionusingfacialdetection-201119063405/85/AI-Therapist-Emotion-Detection-using-Facial-Detection-and-Recognition-and-Showing-Content-According-to-Emotions-1-320.jpg)
![International Journal of Trend in Scientific Research and Development (IJTSRD) @ www.ijtsrd.com eISSN: 2456-6470
@ IJTSRD | Unique Paper ID – IJTSRD33267 | Volume – 4 | Issue – 6 | September-October 2020 Page 30
recognition rate is up to 89%, this is sufficient for assisting
the disabled and their family members or caretakers to use
in their daily life.
Wenlu Yang, Christophe Marsala [4] Investigates the
psychophysiological response to events on an affective
database in a game context. We address some common
challenges for a physiological-based affective model such as
segmentation, normalization, or relevant signals. For
emotion detection task,we show that higharousal emotions
are more detectable than the low arousal ones.
Fig1. Samples of FER 2013 Dataset
III. Emotion Recognition
The process of emotion detection system is to develop a
system that can recognize the faces under different viewing
angles in real time using video in an uncontrolled
environment. The only problem persists while doing this is,
we need to update the output in real time.
Facial detection is important step in emotion detection. The
first thing to do in any machine learning task is to collect the
data. What we need are thousands of images with labelled
facial expressions. The public FER dataset is a great starting
point for that. We imported the FER 2013 dataset for the
emotion detection step.
Fig2. Our Proposed System Architecture
Before we trainan emotion detector, wefirst mustbe ableto
locate the faces within each image. Since the goal is to
classify emotions from facial expressions, it makes sense to
only keep the faces we find and ignore all the irrelevant
things.
A. Converting Images into Grayscale Format
Grayscale isa range of monochromatic shades from black to
white. Therefore, a grayscale image contains only shades of
graycolor and no color. While digital imagescan be savedas
grayscale images, even color images contain somegrayscale
information. This is because each pixel has a certain
brightness or intensity value despite its color. intensity
values which can be measured on a scale from black to
white. Black means zero intensity and White means high
intensity. grayscale images are less complex and having an
only single layer for processing and thus making it efficient
for processing.
B. Histogram equalization
Histogram Equalization is a technique in image processing
for adjusting image intensities to enhance contrast. Some of
our images are blurry, dark and fragile we need to enhance
those images by using Open CV function. For equalizing the
histogram, we need to compute the histogram of the image
and then normalize it into a probability distribution. For
normalization, we just need to divide the frequency of each
pixel intensity value by the total number of pixels present in
the image. This is equal to the resolution of the image. The
equalization process makes sure thattheresultinghistogram
should be flat.
C. Neural Network
Implementation database is categorized as the
Preprocessing phase, Learningphase, validation and testing
phase. This has been carried outusing Keras anditsfunctions
which significantly reduces the time for neural network
training, validation and testing. Keras is a high-level neural
networks API, written in Python and capable of running on
top of Tensor Flow, CNTK, or Theano.
Our Convolutional neural network model consisted oflayers
of Conv2d and Maxpooling parameters. Keras Conv2Dis a
2D Convolution Layer, this layer creates aconvolutionkernel
that isa windwith layersinput which helps produceatensor
of outputs. Max-pooling reduces the dimensionality of
images by reducing the number of pixels in the output from
the previous convolutional layer and it also reduces the
problem of over fitting.
Fig3. Greyscale Conversion
D. Recommendation System
Recommender systems are algorithms that support human
decision making by reducing the amount ofavailableoptions
to choose from. Understanding the factors that influence
decision making is important for designing better
recommender systems. Emotions and personality traits are
psychological constructs that are linked to decision making
and are used to improve recommendations.
The input is acquired in real-time. The hiddenmarkovmodel
classification are used for processing the acquired images.
The frames that are obtained are considered in all frames](https://image.slidesharecdn.com/6aitherapistemotiondetectionusingfacialdetection-201119063405/85/AI-Therapist-Emotion-Detection-using-Facial-Detection-and-Recognition-and-Showing-Content-According-to-Emotions-2-320.jpg)
![International Journal of Trend in Scientific Research and Development (IJTSRD) @ www.ijtsrd.com eISSN: 2456-6470
@ IJTSRD | Unique Paper ID – IJTSRD33267 | Volume – 4 | Issue – 6 | September-October 2020 Page 31
and all pixel formats for the purpose of emotion
classification. The value of each landmark in the face is
calculated and is stored for future use. The efficiency of
classifier is about 90-95%.
The emotions are assigned for every content shown. When
the emotion is transferred the respective content will
displays. There are four emotions that can be used and the
emotions are happy, anger, sad, surprise. When the happy
emotion is recognized the content that are assigned for that
particular emotion are played and the same happens with
the other emotions as well that is it the content is displayed
for the emotions detected respectively.
Fig 4 Recommendation Module
IV. Result
We validated this model in the FER-2013 dataset. This
datasetcontains 35,887grayscale imageswhere each image
belongs to one of the following classes [“angry”, “disgust”,
“fear”, “happy”, “sad”, “surprise”, “neutral”].
Table No.2 Sequential Model Performance
Epochs Loss Accuracy
1 1.4103 0.4147
2 1.1550 0.5257
3 1.0292 0.5817
4 0.9121 0.6352
5 0.8040 0.6867
6 0.6949 0.7329
7 0.5645 0.7840
8 0.4347 0.8373
9 0.3135 0.8875
10 0.2440 0.9150
The system was trained with trainingdata set, and validated
with validating data set to find the best networkarchitecture
and find the optimal solution. Our initial model achieved an
accuracy of 65% in this dataset.
Chart1. Epoch vs Accuracy
Chart2. Epoch vs Loss
V. Conclusion
This paper has described an approach to detect and
recognize face with the CNN and show contents like music,
blog, news according to the emotion of the person and it
describes its overall performance. The algorithm proposed
has three main stages Preprocessing, neural network
classification and testing. Our analysis shows that the
performance ofNeural network data witha preciseaccuracy
rate of 65%. This could significantly use to identify people
suffering from depression, anxiety etc. and at the same time
help them too. If user is often angry, program will suggest
him tips for anger management. Future work also would
investigate the use of a better complementing feature for
real-time classification and recognition and showing the
results live.
VI. References
[1] IliasMaglogiannis, Demosthenes Vouyioukas, Chris
Aggelopoulos, ”Face detection and recognition of
natural human emotion using Markov random fields” ,
Springer-Verlag London Limited 2007.
[2] Octavio Arriaga and Matias Valdenegro, “Real-time
Convolutional Neural Networks for Emotion and
Gender Classification”, Sankt Augustin Germany.
[3] Bee Theng Lau, “Portable real time emotion detection
system for the disabled”, Expert Systems with
Applications.
[4] Wenlu Yang, Christophe Marsala, ”Physiological-Based
Emotion Detection and Recognition in a Video Game
Context”, https://hal.sorbonne-universite.fr/hal-
01784795.](https://image.slidesharecdn.com/6aitherapistemotiondetectionusingfacialdetection-201119063405/85/AI-Therapist-Emotion-Detection-using-Facial-Detection-and-Recognition-and-Showing-Content-According-to-Emotions-3-320.jpg)
AI Therapist – Emotion Detection using Facial Detection and Recognition and Showing Content According to Emotions
- 1. International Journal of Trend in Scientific Research and Development (IJTSRD) Volume 4 Issue 6, September-October 2020 Available Online: www.ijtsrd.com e-ISSN: 2456 – 6470 @ IJTSRD | Unique Paper ID – IJTSRD33267 | Volume – 4 | Issue – 6 | September-October 2020 Page 29 AI Therapist – Emotion Detection using Facial Detection and Recognition & Showing Content According to Emotions Sanket Godbole1, Jaivardhan Shelke2 1Student, 2Software Engineer, 1Departmet Information Technology, Government College of Engineering, Aurangabad, Maharashtra, India 2Kanthaka Consultancy Services Pvt Ltd, Pune, Maharashtra, India ABSTRACT This paper presents an integrated system for emotion detection using facial detection and recognition. we have taken into account the fact that emotions are most widely represented with eyes and mouth expressions. In this research effort, we implement a general convolutional neural network (CNN) building frameworkfor designing real-time CNNs.We validate our models by creating a real-time vision system that accomplishes the tasks of face detection, emotion classification, and generating the content according to the emotion or mood of the person simultaneously in one blended step using our proposed CNN architecture. Our proposed model consisted of modules such as image processing, Feature extraction, feature classification, and recommendation process. The images used in the experiment are pre-processed with various image processing methods like canny edge detection, histogram equalization,fitellipse,andFER dataset ismediated for conducting the experiments. Withatrainedprofilethat can be updated flexibly, a user can detect his/her behavior on a real-time basis. It utilizes the state of the art of face detection and recognition algorithms. KEYWORDS: classification and recognition, Convolutional Neural Networks, Feature Extraction How to cite this paper: Sanket Godbole | Jaivardhan Shelke "AI Therapist – Emotion Detection using Facial Detection and Recognition & Showing Content According to Emotions" Published in International Journal of Trend in Scientific Research and Development (ijtsrd), ISSN: 2456- 6470, Volume-4 | Issue-6, October 2020, pp.29-31, URL: www.ijtsrd.com/papers/ijtsrd33267.pdf Copyright © 2020 by author(s) and International Journal of TrendinScientific Research and Development Journal. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (CC BY 4.0) (http://creativecommons.org/licenses/by /4.0) I. INTRODUCTION Human emotions and its detection and recognition play a very vital role in applications intended for handicapped persons or robotic intelligence or persons with difficulty in recognizing emotions or for developing intelligentGraphical Application. To detect the facial expressions on the human face his face must be detected and the face location and position must be known. Hence in most cases, emotion detection algorithms start with face detection, taking into account the fact that face emotions are mostly depicted in eyes and mouth. There are two mutually benefited subtasks in any facial detection and recognition task that include Detection and Extraction of the image features. facial expression recognition system detects a facewithin the captured image and then it extracts features from the potential face for the recognition task. There are many different typesofemotions that influence how we liveand interactwith others.Attimes, it may seem like we are ruled by these emotions. we intend to find this type of emotion like sadness, anger, happiness, disguise, etc. Furthermore, CNNs are used as black-boxes and often their learned features remain hidden, making it complicated to establisha balance between their classificationaccuracyand unnecessary parameters Therefore, we implemented a real- time visualization. II. Literature Review IliasMaglogiannis, Demosthenes Vouyioukas, and Chris Aggelopoulospresented [1] presented an emotion detection and recognition system for color images. The proposed system detects the skin color using MRFs estimation in YCbCr space, overcoming the difficulty ofvarious skin regions detection under different lighting conditions, emotion recognition then follows, based on the construction of face, eye, and mouth maps. Octavio Arriaga and Matias Valdenegro [2] have proposed and tested general building designs for creating real-time CNNs. Their architecture has been systematically built to reduce the number of parameters. by eliminating the fully connected layers and by reducing the number of parameters in the remaining convolutional layers via depth-wise separable convolutions. Bee Theng Lau [3] developed to help to detect the emotions of the disabled who could not make a speech and also physical movements easily. These peopletendtobeseatedat one fixed location or wheelchair and require assistance in most of the physical movements. the detection and IJTSRD33267
- 2. International Journal of Trend in Scientific Research and Development (IJTSRD) @ www.ijtsrd.com eISSN: 2456-6470 @ IJTSRD | Unique Paper ID – IJTSRD33267 | Volume – 4 | Issue – 6 | September-October 2020 Page 30 recognition rate is up to 89%, this is sufficient for assisting the disabled and their family members or caretakers to use in their daily life. Wenlu Yang, Christophe Marsala [4] Investigates the psychophysiological response to events on an affective database in a game context. We address some common challenges for a physiological-based affective model such as segmentation, normalization, or relevant signals. For emotion detection task,we show that higharousal emotions are more detectable than the low arousal ones. Fig1. Samples of FER 2013 Dataset III. Emotion Recognition The process of emotion detection system is to develop a system that can recognize the faces under different viewing angles in real time using video in an uncontrolled environment. The only problem persists while doing this is, we need to update the output in real time. Facial detection is important step in emotion detection. The first thing to do in any machine learning task is to collect the data. What we need are thousands of images with labelled facial expressions. The public FER dataset is a great starting point for that. We imported the FER 2013 dataset for the emotion detection step. Fig2. Our Proposed System Architecture Before we trainan emotion detector, wefirst mustbe ableto locate the faces within each image. Since the goal is to classify emotions from facial expressions, it makes sense to only keep the faces we find and ignore all the irrelevant things. A. Converting Images into Grayscale Format Grayscale isa range of monochromatic shades from black to white. Therefore, a grayscale image contains only shades of graycolor and no color. While digital imagescan be savedas grayscale images, even color images contain somegrayscale information. This is because each pixel has a certain brightness or intensity value despite its color. intensity values which can be measured on a scale from black to white. Black means zero intensity and White means high intensity. grayscale images are less complex and having an only single layer for processing and thus making it efficient for processing. B. Histogram equalization Histogram Equalization is a technique in image processing for adjusting image intensities to enhance contrast. Some of our images are blurry, dark and fragile we need to enhance those images by using Open CV function. For equalizing the histogram, we need to compute the histogram of the image and then normalize it into a probability distribution. For normalization, we just need to divide the frequency of each pixel intensity value by the total number of pixels present in the image. This is equal to the resolution of the image. The equalization process makes sure thattheresultinghistogram should be flat. C. Neural Network Implementation database is categorized as the Preprocessing phase, Learningphase, validation and testing phase. This has been carried outusing Keras anditsfunctions which significantly reduces the time for neural network training, validation and testing. Keras is a high-level neural networks API, written in Python and capable of running on top of Tensor Flow, CNTK, or Theano. Our Convolutional neural network model consisted oflayers of Conv2d and Maxpooling parameters. Keras Conv2Dis a 2D Convolution Layer, this layer creates aconvolutionkernel that isa windwith layersinput which helps produceatensor of outputs. Max-pooling reduces the dimensionality of images by reducing the number of pixels in the output from the previous convolutional layer and it also reduces the problem of over fitting. Fig3. Greyscale Conversion D. Recommendation System Recommender systems are algorithms that support human decision making by reducing the amount ofavailableoptions to choose from. Understanding the factors that influence decision making is important for designing better recommender systems. Emotions and personality traits are psychological constructs that are linked to decision making and are used to improve recommendations. The input is acquired in real-time. The hiddenmarkovmodel classification are used for processing the acquired images. The frames that are obtained are considered in all frames
- 3. International Journal of Trend in Scientific Research and Development (IJTSRD) @ www.ijtsrd.com eISSN: 2456-6470 @ IJTSRD | Unique Paper ID – IJTSRD33267 | Volume – 4 | Issue – 6 | September-October 2020 Page 31 and all pixel formats for the purpose of emotion classification. The value of each landmark in the face is calculated and is stored for future use. The efficiency of classifier is about 90-95%. The emotions are assigned for every content shown. When the emotion is transferred the respective content will displays. There are four emotions that can be used and the emotions are happy, anger, sad, surprise. When the happy emotion is recognized the content that are assigned for that particular emotion are played and the same happens with the other emotions as well that is it the content is displayed for the emotions detected respectively. Fig 4 Recommendation Module IV. Result We validated this model in the FER-2013 dataset. This datasetcontains 35,887grayscale imageswhere each image belongs to one of the following classes [“angry”, “disgust”, “fear”, “happy”, “sad”, “surprise”, “neutral”]. Table No.2 Sequential Model Performance Epochs Loss Accuracy 1 1.4103 0.4147 2 1.1550 0.5257 3 1.0292 0.5817 4 0.9121 0.6352 5 0.8040 0.6867 6 0.6949 0.7329 7 0.5645 0.7840 8 0.4347 0.8373 9 0.3135 0.8875 10 0.2440 0.9150 The system was trained with trainingdata set, and validated with validating data set to find the best networkarchitecture and find the optimal solution. Our initial model achieved an accuracy of 65% in this dataset. Chart1. Epoch vs Accuracy Chart2. Epoch vs Loss V. Conclusion This paper has described an approach to detect and recognize face with the CNN and show contents like music, blog, news according to the emotion of the person and it describes its overall performance. The algorithm proposed has three main stages Preprocessing, neural network classification and testing. Our analysis shows that the performance ofNeural network data witha preciseaccuracy rate of 65%. This could significantly use to identify people suffering from depression, anxiety etc. and at the same time help them too. If user is often angry, program will suggest him tips for anger management. Future work also would investigate the use of a better complementing feature for real-time classification and recognition and showing the results live. VI. References [1] IliasMaglogiannis, Demosthenes Vouyioukas, Chris Aggelopoulos, ”Face detection and recognition of natural human emotion using Markov random fields” , Springer-Verlag London Limited 2007. [2] Octavio Arriaga and Matias Valdenegro, “Real-time Convolutional Neural Networks for Emotion and Gender Classification”, Sankt Augustin Germany. [3] Bee Theng Lau, “Portable real time emotion detection system for the disabled”, Expert Systems with Applications. [4] Wenlu Yang, Christophe Marsala, ”Physiological-Based Emotion Detection and Recognition in a Video Game Context”, https://hal.sorbonne-universite.fr/hal- 01784795.