![Literature Review
According to literatures, it’s wildly known that there are six different types of emotion:
Happiness, Anger, Sadness, Fear, Surprise and Disgust. Besides, neutral is another option as a
seventh emotion in some literature for emotion recognition. This thesis concentrates on
identifying the facial expression using thermal images instead of visible image recognition.
1.1 Visible image based spontaneous expression recognition
For the past few years, with the gradual development of imaging processing technology,
facial expression recognition of visible images is rapidly developed. The main research
content include following fields:
Face Detection, Location and Tracking, determine the face location among a variety of
images, extract features then track the certain face in a bunch of image sequence. This step
named as the pre-process of expression recognition has become an independent research
topic with serious treatment.
Feature Extraction, extract relative information to characterize facial emotion among a set
of facial images or image sequence. This feature can be captured in time-domain or
frequency-domain which contains the gray information, geometric feature or texture
information etc. Feature extraction is the key step and difficulty of entire facial emotion
recognition, meanwhile, appropriate features can promote the efficiency and effect of the
classification system. So far, the method of feature extraction include: using Karhunen-
Loeve procedure[1], PCA[2], 2D DCT[3] and k-means algorithm, Singular Value
Decomposition and Hidden Markov Model, to capture expression feature vectors.
Expression Recognition, adopts classifier and classification algorithms of pattern
recognition system which inputs are a set of image futures and outputs are belonged to an
emotion catalogue. In order to determine the best algorithm and classifier, the method
wildly used are KNN algorithms; SVM expression classifier[4]; Facial Animation
Parameters and Multistream HMMs[5] [6]etc.
Based on the general research of emotion recognition, numerous constructive](https://image.slidesharecdn.com/0a3dc192-12d5-4883-8e10-728b6f959a0a-160629053442/85/Thermal-Imaging-Emotion-Recognition-final-report-01-4-320.jpg)
![methodologies were proposed to identify human’s facial expression with high accuracy
and efficiency. However, most of them focus on posted facial expression to distinguish the
emotion. On the contrary, expression is a kind of facial movement inspired by inner feeling
in real life. Therefore, spontaneous expressions can better reflect the authentic thought of a
person when compared with posted ones which maintain more valuable meaning.
Consequently, this thesis aims on researching spontaneous expression recognition.
1.2 The research status of infrared image expression recognition
Up to now, the emotion recognition achieves a certain progress mainly in the field of
visible images or image sequences. Whereas, because of the imaging mechanism of
visible images, the facial expression recognition algorithms are interfered by illumination
situation based on this field. However, what thermal cameras capture is temperature
distribution of facial vein branches which is not sensitive to lightning conditions.
Therefore, the thesis topic this report focuses on can remedy the disadvantages and faults
of visible image emotion recognition to a great extent which is a significant research trend
in future[7].
Currently, the research of thermal emotion recognition is just getting started in both posted
and spontaneous expression. Representative researches among them are:
1. The methodology of image segmentation by Yoshiaki Sugimoto is divided face into
eyes, nose, mouse, cheek, chin etc. to extract feature of each part then fit a straight
line in order to analyzing the difference of infrared image thermal field distribution.
However, this analytical method depends on thermal images with high degree of
accuracy and high requirement of photographic equipment[8].
2. Masood Mehmood Khan came up with a classification method in three-dimension
domain. In this paper, it combines LDA taxonomy with testing temperature of certain
facial points to establish the 3D domain model[9]. In addition, Leonardo Trujillo
combines SVM taxonomy with facial point temperature to classify emotion as
well[10]. These two methodologies perform well but with limitation of professional
equipment to measure facial point temperature which is a weakness of real-time
system processing.
3. Y. Yoshitomi mentioned that process the subtraction operation between expression
images and neutral images firstly, segment to extract features then combines the](https://image.slidesharecdn.com/0a3dc192-12d5-4883-8e10-728b6f959a0a-160629053442/85/Thermal-Imaging-Emotion-Recognition-final-report-01-5-320.jpg)
![neural network learning to classify human emotion[11]. Compared this method with
previous ones, it avoids processing many unnecessary data to increase the efficiency
of algorithms and recognition rate ( the recognition rate for Neutral, Happy, Surprise,
Sad expressions are 80%, 95%, 100%, 85% separately). Nevertheless, when
subtracting with neutral images, it inevitably involves image alignment problem
which may introduce error during the whole actual operation.
4. Guotai Jiang and other research stuffs think using mathematical morphology to
extract features and classify emotion. This method is operated in low dimension and
easy to achieve[12]. However, there are only two pictures being used in experiment as
database, thus, the correctness and extensibility of this algorithm remain to be
discussed.
5. In Benjamin Hernandez’s theory, extracting features of forehead, eyes, cheeks, mouse
to analysis with SVM taxonomy is an effective way to reduce the amount of data
which the recognition rate is around 76.6%[13]. But, it’s hard to solve how to
normalize the segmented regions in real operation.
6. Jenkins reports using Pearson product-moment relative coefficients to calculate the
correlation between temperature change of forehead and EEG self-report. It indicates
that the forehead temperature variation can reflect different emotion of human
beings[14].
7. In Jarlier’s report, it analysis the difference in thermal images when facial motion
units move in various strength and speed. In the experiment, 4 experts play 9 different
facial movements to test which proves the feasibility of thermal analysis of facial
muscles contraction[15]. However, how to tract the movement unit is an obstacle of
this technology.
In this report, the method is extracting facial thermal features based on the wavelet
analysis. According to the advantage of this method, it is not necessary to track and
segment face image or subtract the image registration which reduce the error during
image registration. Meanwhile, as a methodology of gray scale matrix conversion, it can
contain more valuable information of thermal images. In this report, it proposed a
method of infrared facial expression recognition based on 2 layers 2D wavelet transform.
It does 2 layers of 2D wavelet transform to images, chooses approximation coefficients
as features by Largest Euclidean Distance, and recognizes the expressions by K -Nearest
Neighbors. The algorithm is validated effect on USTC-NVIE database.](https://image.slidesharecdn.com/0a3dc192-12d5-4883-8e10-728b6f959a0a-160629053442/85/Thermal-Imaging-Emotion-Recognition-final-report-01-6-320.jpg)
![[1] S. M Irby, “Application of the Karhunen-Loeve procedure for the characterization of
human face,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol.
12(1), pp. 103-108, 1990.
[2] D. R. H. E. S. D, “Pattern Classification( second edition),” New York: Wiley-
Interscience, pp. 114-139, 2000.
[3] L. X. Ma, Y Khorasani, K. Ward, R.K., “A new facial expression recognition
technique using 2D DCT and k-means algorithm,”
Image Processing, vol. 2, pp. 1269-1272, 2004.
[4] F. Y. S. Chao-Fa Chuang, “Recognizing facial action units independent component
analysis and support vector machine,” Pattern Recognition, vol. 39, pp. 1795-1798,
2006.
[5] G. A. F. Thomas, “Pattern recognition methods for advanced stochastic protein
sequence analysis using HMMs,” Pattern Recognition, vol. 39, pp. 2267-2280, 2006.
[6] A. K. K. Petar S. Aleksic, “Facial Expression Recognition Using Facial Animation
Aprameters and Multistream HMMs,” IEEE Transactions on Information Forensics
and Security, vol. 1, no. 1, pp. 3-11, 2006.
[7] S. L. Shangfei Wang, Xufa Wang, “Infrared Facial Expression Recognition Using
Wavelet Transform,” in International Symposium on Computer Science and
Computational Technology, 2008.
[8] T. Y. Yoshiaki Sugimoto, Shigeyuki Tomita, “A method for detecting transitions of
emotional stated using thermal facial image based on a synthesis of facial
expressions,” Robotics and Autonomous Systems, vol. 31, no. 2000, pp. 147-160,
2000.
[9] R. D. W. Masood Mehmood Khan, Michael Ingleby, "Automated classification and
recognition of facial expressions using infrared thermal imaging."
[10] G. O. Leonardo Trujillo, Riad Hammoud and Benjamin Hernandez, “Automatic
Feature Localization in Thermal Images for Facial Expression Recognition.”
[11] N. M. Y. Yoshitomi, S. Tomita and S. Kimura, “Facial Expression Recognition Using
Thermal Image Processiong and Neural Network,” 1997.
[12] X. S. Guotai Jiang, Fuhui Zheng, Peipei Wang, Ashgan M. Omer, "Facial Expression
Recognition Using Thermal Images."
[13] B. H. n. e. al, “Visual learning of texture descriptors for facial expression recognition
in thermal imagery,” Image Understand, 2007.](https://image.slidesharecdn.com/0a3dc192-12d5-4883-8e10-728b6f959a0a-160629053442/85/Thermal-Imaging-Emotion-Recognition-final-report-01-7-320.jpg)
![[14] S. Jenkins, Brown, R., and Rutterford, N, “Comparing Thermographic, EEG and
Subjective Measures of Affective Experience During Simulated Product Interactions,”
International Journal of Design, vol. 3, no. 2, pp. 52-65, 2009.
[15] S. Jarlier, Grandjean, D., Delplanque, S., N'Diaye, K., Cayeux,I., Velazco, N.,,
“Thermal Analysis of Facial Muscles Contractions,” IEEE Transactions vol. 1, no. 1,
pp. 99.](https://image.slidesharecdn.com/0a3dc192-12d5-4883-8e10-728b6f959a0a-160629053442/85/Thermal-Imaging-Emotion-Recognition-final-report-01-8-320.jpg)
Thermal Imaging Emotion Recognition final report 01
- 1. Thermal Imaging Emotion Recognition A thesis submitted in partial fulfillment of the requirements for the award of the degree Bachelor of Engineering (Telecommunication) From University of Wollongong by AI ZHANG School of Electrical, Computer and Telecommunications Engineering June, 2013 Supervisor: DR S. L. PHUNG
- 2. STATEMENT OF ORIGINALITY I, Ai Zhang, declares that this thesis, submitted as part of the requirements for the award of Bachelor of Engineering, in the School of Electrical, Computer and Telecommunications Engineering, University of Wollongong, is wholly my own work unless otherwise referenced or acknowledged. The document has not been submitted for qualifications or assessment at any other academic institution. Signature: ………………………………… Print Name: ………………………………… Student ID Number: ………………………………… Date: …………………………………
- 3. Abbreviations and Symbols NVIE Natural visible and infrared facial expression database PCA Principle component analysis LDA Linear discriminant analysis AAM Active Appearance Model HE Histogram equalization RHE Regional histogram equalization GT Gamma transformation RGT Regional gamma transformation DSP Digital Signal Processor USB Universal Serial Bus AVI Audio Video Interlace K-L Karhunen-Loeve 2D DWT
- 4. Literature Review According to literatures, it’s wildly known that there are six different types of emotion: Happiness, Anger, Sadness, Fear, Surprise and Disgust. Besides, neutral is another option as a seventh emotion in some literature for emotion recognition. This thesis concentrates on identifying the facial expression using thermal images instead of visible image recognition. 1.1 Visible image based spontaneous expression recognition For the past few years, with the gradual development of imaging processing technology, facial expression recognition of visible images is rapidly developed. The main research content include following fields: Face Detection, Location and Tracking, determine the face location among a variety of images, extract features then track the certain face in a bunch of image sequence. This step named as the pre-process of expression recognition has become an independent research topic with serious treatment. Feature Extraction, extract relative information to characterize facial emotion among a set of facial images or image sequence. This feature can be captured in time-domain or frequency-domain which contains the gray information, geometric feature or texture information etc. Feature extraction is the key step and difficulty of entire facial emotion recognition, meanwhile, appropriate features can promote the efficiency and effect of the classification system. So far, the method of feature extraction include: using Karhunen- Loeve procedure[1], PCA[2], 2D DCT[3] and k-means algorithm, Singular Value Decomposition and Hidden Markov Model, to capture expression feature vectors. Expression Recognition, adopts classifier and classification algorithms of pattern recognition system which inputs are a set of image futures and outputs are belonged to an emotion catalogue. In order to determine the best algorithm and classifier, the method wildly used are KNN algorithms; SVM expression classifier[4]; Facial Animation Parameters and Multistream HMMs[5] [6]etc. Based on the general research of emotion recognition, numerous constructive
- 5. methodologies were proposed to identify human’s facial expression with high accuracy and efficiency. However, most of them focus on posted facial expression to distinguish the emotion. On the contrary, expression is a kind of facial movement inspired by inner feeling in real life. Therefore, spontaneous expressions can better reflect the authentic thought of a person when compared with posted ones which maintain more valuable meaning. Consequently, this thesis aims on researching spontaneous expression recognition. 1.2 The research status of infrared image expression recognition Up to now, the emotion recognition achieves a certain progress mainly in the field of visible images or image sequences. Whereas, because of the imaging mechanism of visible images, the facial expression recognition algorithms are interfered by illumination situation based on this field. However, what thermal cameras capture is temperature distribution of facial vein branches which is not sensitive to lightning conditions. Therefore, the thesis topic this report focuses on can remedy the disadvantages and faults of visible image emotion recognition to a great extent which is a significant research trend in future[7]. Currently, the research of thermal emotion recognition is just getting started in both posted and spontaneous expression. Representative researches among them are: 1. The methodology of image segmentation by Yoshiaki Sugimoto is divided face into eyes, nose, mouse, cheek, chin etc. to extract feature of each part then fit a straight line in order to analyzing the difference of infrared image thermal field distribution. However, this analytical method depends on thermal images with high degree of accuracy and high requirement of photographic equipment[8]. 2. Masood Mehmood Khan came up with a classification method in three-dimension domain. In this paper, it combines LDA taxonomy with testing temperature of certain facial points to establish the 3D domain model[9]. In addition, Leonardo Trujillo combines SVM taxonomy with facial point temperature to classify emotion as well[10]. These two methodologies perform well but with limitation of professional equipment to measure facial point temperature which is a weakness of real-time system processing. 3. Y. Yoshitomi mentioned that process the subtraction operation between expression images and neutral images firstly, segment to extract features then combines the
- 6. neural network learning to classify human emotion[11]. Compared this method with previous ones, it avoids processing many unnecessary data to increase the efficiency of algorithms and recognition rate ( the recognition rate for Neutral, Happy, Surprise, Sad expressions are 80%, 95%, 100%, 85% separately). Nevertheless, when subtracting with neutral images, it inevitably involves image alignment problem which may introduce error during the whole actual operation. 4. Guotai Jiang and other research stuffs think using mathematical morphology to extract features and classify emotion. This method is operated in low dimension and easy to achieve[12]. However, there are only two pictures being used in experiment as database, thus, the correctness and extensibility of this algorithm remain to be discussed. 5. In Benjamin Hernandez’s theory, extracting features of forehead, eyes, cheeks, mouse to analysis with SVM taxonomy is an effective way to reduce the amount of data which the recognition rate is around 76.6%[13]. But, it’s hard to solve how to normalize the segmented regions in real operation. 6. Jenkins reports using Pearson product-moment relative coefficients to calculate the correlation between temperature change of forehead and EEG self-report. It indicates that the forehead temperature variation can reflect different emotion of human beings[14]. 7. In Jarlier’s report, it analysis the difference in thermal images when facial motion units move in various strength and speed. In the experiment, 4 experts play 9 different facial movements to test which proves the feasibility of thermal analysis of facial muscles contraction[15]. However, how to tract the movement unit is an obstacle of this technology. In this report, the method is extracting facial thermal features based on the wavelet analysis. According to the advantage of this method, it is not necessary to track and segment face image or subtract the image registration which reduce the error during image registration. Meanwhile, as a methodology of gray scale matrix conversion, it can contain more valuable information of thermal images. In this report, it proposed a method of infrared facial expression recognition based on 2 layers 2D wavelet transform. It does 2 layers of 2D wavelet transform to images, chooses approximation coefficients as features by Largest Euclidean Distance, and recognizes the expressions by K -Nearest Neighbors. The algorithm is validated effect on USTC-NVIE database.
- 7. [1] S. M Irby, “Application of the Karhunen-Loeve procedure for the characterization of human face,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 12(1), pp. 103-108, 1990. [2] D. R. H. E. S. D, “Pattern Classification( second edition),” New York: Wiley- Interscience, pp. 114-139, 2000. [3] L. X. Ma, Y Khorasani, K. Ward, R.K., “A new facial expression recognition technique using 2D DCT and k-means algorithm,” Image Processing, vol. 2, pp. 1269-1272, 2004. [4] F. Y. S. Chao-Fa Chuang, “Recognizing facial action units independent component analysis and support vector machine,” Pattern Recognition, vol. 39, pp. 1795-1798, 2006. [5] G. A. F. Thomas, “Pattern recognition methods for advanced stochastic protein sequence analysis using HMMs,” Pattern Recognition, vol. 39, pp. 2267-2280, 2006. [6] A. K. K. Petar S. Aleksic, “Facial Expression Recognition Using Facial Animation Aprameters and Multistream HMMs,” IEEE Transactions on Information Forensics and Security, vol. 1, no. 1, pp. 3-11, 2006. [7] S. L. Shangfei Wang, Xufa Wang, “Infrared Facial Expression Recognition Using Wavelet Transform,” in International Symposium on Computer Science and Computational Technology, 2008. [8] T. Y. Yoshiaki Sugimoto, Shigeyuki Tomita, “A method for detecting transitions of emotional stated using thermal facial image based on a synthesis of facial expressions,” Robotics and Autonomous Systems, vol. 31, no. 2000, pp. 147-160, 2000. [9] R. D. W. Masood Mehmood Khan, Michael Ingleby, "Automated classification and recognition of facial expressions using infrared thermal imaging." [10] G. O. Leonardo Trujillo, Riad Hammoud and Benjamin Hernandez, “Automatic Feature Localization in Thermal Images for Facial Expression Recognition.” [11] N. M. Y. Yoshitomi, S. Tomita and S. Kimura, “Facial Expression Recognition Using Thermal Image Processiong and Neural Network,” 1997. [12] X. S. Guotai Jiang, Fuhui Zheng, Peipei Wang, Ashgan M. Omer, "Facial Expression Recognition Using Thermal Images." [13] B. H. n. e. al, “Visual learning of texture descriptors for facial expression recognition in thermal imagery,” Image Understand, 2007.
- 8. [14] S. Jenkins, Brown, R., and Rutterford, N, “Comparing Thermographic, EEG and Subjective Measures of Affective Experience During Simulated Product Interactions,” International Journal of Design, vol. 3, no. 2, pp. 52-65, 2009. [15] S. Jarlier, Grandjean, D., Delplanque, S., N'Diaye, K., Cayeux,I., Velazco, N.,, “Thermal Analysis of Facial Muscles Contractions,” IEEE Transactions vol. 1, no. 1, pp. 99.