SSHOC kick off meeting - 1.1.1 Where Social Sciences and Humanities Meet: Investigating emotional expressions of people with dementia - Khiet Truong - University of Twente
1 like•95 views
The document discusses a research project investigating the emotional expressions of elderly individuals with dementia, focusing on both verbal and nonverbal recognition of these emotions. It details the creation of a multimodal corpus that includes emotional expressions from both healthy elderly and those with dementia, with the aim to develop emotion recognition models. The study also highlights the importance of refining emotion coding schemes and addressing the challenges of eliciting genuine emotional responses.
SSHOC kick off meeting - 1.1.1 Where Social Sciences and Humanities Meet: Investigating emotional expressions of people with dementia - Khiet Truong - University of Twente
- 1. Investigating emotional expressions of people with dementia Khiet Truong, Dirk Heylen Deniece Nazareth, Gerben Westerhof Human Media Interaction Psychology, Health & Technology
- 3. Team UT: Dept. of Psychology, Health & Technology Prof.dr. Gerben Westerhof Deniece Nazareth, MSc UT: Human Media Interaction Prof.dr. Dirk Heylen Dr. Khiet Truong Deniece Nazareth, MSc UMCG Dr. Miriam Kunz eScience Center Amsterdam Dr. Ben de Vries Dr. Vincent van Hees 3
- 4. Emotional Expressions of People with Dementia • How does dementia influence emotional expression? • Can we automatically recognize these emotional expressions through verbal (words) and nonverbal (voice, face) information?
- 5. Beyond state-of-the-art • Some is known about how elderly recognize emotion (how does dementia affect how people perceive emotion) ➤ Learn more about spontaneous emotion production in healthy elderly and elderly with dementia • There is a lot of research about automatic emotion recognition in healthy adults ➤ Investigate and develop emotion recognition models and tools for healthy elderly and elderly with dementia ➤ Develop a publicly available multimodal corpus with spontaneous emotional expressions of elderly 5
- 6. Multimodal corpus of emotional expressions Phase 1: healthy elderly Phase 2: elderly with dementia + control group (matched healthy elderly) 6 ✓
- 7. 7 Lifestory books & Reminiscence Therapy Recalling past memories Visit 1 Autobiographical memory test (AMT) Visit 2 Life story book
- 8. Visit 1: Autobiographical memory test (Williams & Broadbent, 1986) • Recalling specific autobiographical memories - A specific event that happened once in a specific place on a specific date that did not last longer than a day 8 Autobio- graphical memory test (AMT) Word 1 Happy “I received my diploma in March 2014 at the university and my grandmother was there.”
- 9. Visit 1: Autobiographical memory test (Williams & Broadbent, 1986) • Recalling specific autobiographical memories - A specific event that happened once in a specific place on a specific date that did not last longer than a day 9 Autobio- graphical memory test (AMT) Word 1 Happy “I have always been a happy kid.”
- 10. Visit 2: Lifestory book 10 “Could you tell me more about this?” “What were you feeling back then?” “How did people around you react?” “How was life for you after this happened?”
- 11. Visit 2: Lifestory book - SAM How did you feel at that moment when it happened? How did you feel when you told me about the event? How are you feeling now? 11
- 12. Visit 2: IAPS pictures 12
- 13. Procedure 13 Informed Consent Personality Test AMT MOCA V1 Demographic Information Lifestory book IAPS V2
- 14. 14 “Well, after giving birth, eh, she did not cry and it scared me, but it is normal, but I did not know that yet. So she was picked up by her legs and she got a slap on her butt and then she started to cry of course.”
- 15. 15 Participants 23 participants: 65 – 85 years old 11 male, 12 female 22 born in Netherlands, 1 born in Indonesia All are living in region of Twente Recruited via regional newspaper Data V1: ~1 hour (audio only) V2: ~1,5 hours (audiovisual + heart rate + electrodermal activity + accelerometer)
- 16. Beyond state-of-the-art ➤ Learn more about spontaneous emotion production in healthy elderly and elderly with dementia ➤ Investigate and develop emotion recognition models and tools for healthy elderly and elderly with dementia ➤ Develop a publicly available multimodal corpus with spontaneous emotional expressions of elderly 16
- 17. Processing and Analyses 1. When is the speaker emotional, and is this a positive or negative emotion? (labelling task) 2. What verbal and nonverbal features to extract to recognize these emotions automatically? 3. How can we make this data (and tools) available and useful for scholars both in computer science and social sciences / humanities as well? 17
- 18. 2. Can we use verbal and nonverbal features to recognize these emotions automatically? 18 Word-level transcriptions: Dutch ASR Kaldi Prosody: Praat Facial expressions: openFace
- 19. 3. How can we make this data (and tools) available and useful for scholars both in computer science and social sciences / humanities as well? 19
- 20. 1. When is the speaker emotional, and is this a positive or negative emotion? (labelling task) • Challenging: labelling spontaneous emotions is not straightforward! • Sad memory: Her divorce felt 2-sided – disappointment but it also set her free 20
- 22. Lessons learned so far • Elicitation via IAPS pictures gives rather artificial results • Blended emotions • This natural setting of recalling memories calls for novel coding schemes of emotion 22
- 23. Next steps • Developing a coding scheme for valence (positive vs negative emotions) - Taking into account that these are memories, lifestories - Taking into account that emotions can be blended • How to make data accessible for other researchers? 23
- 24. Khiet Truong, Dirk Heylen {k.p.truong, d.k.j.heylen}@utwente.nl http://hmi.ewi.utwente.nl Human Media Interaction Psychology, Health & Technology Thank you