Making an on-device personal assistant a reality
Download as PPTX, PDF2 likes448 views
The document discusses the development of on-device AI personal assistants, emphasizing advancements in voice user interfaces (UI) powered by machine learning. Key features include efficient model design, robust personalization, and real-time natural language understanding for improved user interaction. The integration of various AI capabilities aims to create a truly responsive and proactive virtual assistant that can operate with minimal data while preserving user privacy.
Making an on-device personal assistant a reality
- 1. Making an on-device personal assistant a reality Jilei Hou Sr. Director, Engineering Qualcomm Technologies, Inc. @qualcomm_techJuly 10, 2018 San Diego, CA
- 2. 2 Reasoning Learn, infer context, and anticipate Perception Hear, see, and observe Action Act intuitively, interact naturally, and protect privacy AI brings human-like understanding and behaviors to the machines
- 3. 3 Advancing AI research to make on-device AI ubiquitous A common platform is fundamental to scaling AI internally and across the industry Power efficiency Model design, compression, quantization, activation, algorithms, and efficient hardware Efficient learning Robust learning through minimal data, unsupervised learning, and on-device learning Personalization Continuous learning, model adaptation, and privacy-preserved distributed learning System architecture Multi-task and multi-modal learning, sensor fusion, and cloud-edge systems Action Reinforcement learning for decision making Reasoning Scene understanding, language understanding, behavior prediction Perception Object detection, speech recognition, contextual fusion AutomotiveIoT Mobile
- 4. 4 A true personal assistant One of many use cases requiring a broad set of AI capabilities Power efficiency Model design, compression, quantization, activation, algorithms, and efficient hardware Efficient learning Robust learning through minimal data, unsupervised learning, and on-device learning Personalization Continuous learning, model adaptation, and privacy-preserved distributed learning AutomotiveIoT Mobile Action Reinforcement learning for decision making Reasoning Scene understanding, language understanding, behavior prediction Perception Object detection, speech recognition, contextual fusion System architecture Multi-task and multi-modal learning, sensor fusion, and cloud-edge systems
- 5. 5 v Designed to be: Always-on Conversational Personal Private Critical to create a true virtual assistant Voice is the transformative user interface (UI) we’ve been waiting for
- 6. 6 Voice UI components required for an end-to-end solution Text-to-speech Speech synthesis Natural language generation Dialog management Natural language understanding Natural language processing Speech-to-text Speech recognition “Alexa,” “Hey Snapdragon” Always-on keyword detection Voice activation Echo cancellation Speech denoising Speech pre-processing Signal acquisition and playback Front-end processing Machine speech chain: listener and speaker Qualcomm Snapdragon is a product of Qualcomm Technologies, Inc. and/or its subsidiaries.
- 7. 77 Machine learning has ignited the voice UI revolution “As speech recognition accuracy goes from say 95% to 99%, all of us in the room will go from barely using it today to using it all the time. Most people underestimate the difference between 95% and 99% accuracy— 99% is a gamechanger. No one wants to wait 10 seconds for a response. Accuracy, followed by latency, are the two key metrics for a production speech system.” — Andrew Ng GMM: Gaussian Mixture Model, CNN: Convolutional Neural Network, RNN: Recurrent Neural Network Human accuracy GMM RNN + CNN Machine automatic speech recognition accuracy 202020102000199019801970 50% CNN 55% 60% 62% 70% 95%
- 8. 8 TV Smart speakers Headphones and headsets IoT Smartphones and tablets In-car entertainment systems PCs and laptops Wearables XR Voice UI is proliferating across product categories
- 9. 9 Moving voice UI functionality to the end device An end-to-end solution powered by machine learning Automatic speech recognition (ASR) Text-to-speech (TTS) News SMS Music Maps Wikipedia Weather Stocks Voice activation Service manager Multi-mic echo cancellation, beamforming, and speech denoising Natural language understanding (NLU) On-device processing (always-on and real-time) Cloud processing (services) Cloud centric (today)
- 10. 10 Moving voice UI functionality to the end device An end-to-end solution powered by machine learning Automatic speech recognition (ASR) Text-to-speech (TTS) News SMS Music Maps Wikipedia Weather Stocks Voice activation Service manager Multi-mic echo cancellation, beamforming, and speech denoising Natural language understanding (NLU) On-device processing (always-on and real-time) Cloud processing (services) On-device centric (future)
- 11. 11 Cloud tasks Complex voice fallback Training and model update Knowledge base Services On-device processing of voice UI Provides unique benefits complementing the cloud Challenge Providing the voice UI functionality within the power/thermal envelope Machine learning models Offline raw data Queries On-device tasks Automatic speech recognition Natural language processing Always-on audio cognition On-device training Benefits Privacy Instant response Always-on Device context
- 12. 12 Noisy speech spectrogram Clean speech spectrogram “If people were more generous, there would be no need for welfare” 0.5 1 1.5 2 2.5 3 4000 3000 2000 1000 0 Frequency Time 0.5 1 1.5 2 2.5 3 4000 3000 2000 1000 0 Frequency Time DL-based denoising model trained with extensive speech noise databases DL-based denoising Speech denoising • Single or multiple mics • Applicable for ◦ Two-way conversation ◦ Voice/speaker recognition ◦ Keyword spotting • Deep learning (DL) significantly improves the performance over traditional methods • Robust in challenging interference and noise scenarios “If people were more generous, there would be no need for welfare”
- 13. 13 Qualcomm Voice Activation supports: Qualcomm® Voice Activation (VA) High accuracy, robust to background noise, and supports multiple languages Deep learning is improving performance Among state-of-the-art in terms of performance vs. power consumption -47% -11% 2014 2015 2016 2017 QualcommVApowerconsumption WCD9330 WCD9335 WCD9340 Amazon Alexa Baidu DUEROS Microsoft Cortana Google Assistant Qualcomm Voice Activation, Qualcomm WCD9330, Qualcomm WCD9335, and Qualcomm WCD9340 are products of Qualcomm Technologies, Inc. and/or its subsidiaries.
- 14. 14 Transcribe the audio to text Deep learning gives state-of-the-art accuracy on a mobile device Personalization—adaptation to individual accent and acoustic environment Automatic speech recognition Natural language understanding (NLU) Acoustic features Acoustic model Reduce input audio to essential information Language model Deep learning converts input into linguistic units Adapted to each user’s accent and environment Uses context and language statistics for best utterance estimation Adapted to each user’s speaking tendencies Allows the same intention to be expressed in multiple ways Adapted to each user’s intent expressions “Turn on the light” On-device automatic speech recognition (ASR) User intention
- 15. 1515 An end-to-end on-device voice UI example for smart homes 99% on-device intent accuracy is achieved for domain specific command sets when adapted to accent and environmental condition Demo of automatic speech recognition and natural language understanding Large command set Turn on the living room lights Click the kitchen lights off Turn off all lights Switch on the ceiling fan Shut off the sprinklers Start music Pause song Next track Go back one Play previous song Turn speaker off Increase temperature Intent understanding Turn on the kitchen light Click kitchen light on Switch on light in the kitchen Turn the light on in the kitchen NLU: These four phrases map to the same intent
- 16. 16 A true virtual assistant A “digital me” sitting on the device: context aware and personalized
- 17. 1717 Calendar Messaging Apps On-device data Contextual intelligence is required for personalization The fusion of many types of sensors and personal information 17 Low power sensing, processing, and connectivity Efficient, heterogeneous architectures Sensor fusion and machine learning Integrated, always-on data capturing Low-energy wireless technologies (e.g. BT-LE, 5G NR IoT) Cloud data Off-device data IoT data Sensor fusion Gyroscope Compass Camera Ambient light Temperature Iris scanEnvironment Pulse HumidityMicrophone Sensor data C-V2X
- 18. 18 Various kitchen noise samples Posterior-gram ML-based acoustic event classification Object rustling Object snapping Cupboard Cutlery Dishes Drawer Glass jingling Object impact Walking Washing dishes Water tap running 2 4 6 8 10 12 50 100 150 200 250 300 350 400 450 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 Acoustic event detection • ML techniques are used to ◦ Classify acoustic signals into a set of predefined events ◦ Infer acoustic environment • Low power, always-on
- 19. 1919 Creating personalized memories Essential for a true virtual assistant History, number of people, identity After the party, strolling on the beach at sunset in La Jolla talking with my son and laughing Live sentiment analysis Strolling on the beach at sunset in La Jolla talking with my son and laughing Activity analysis Strolling on the beach at sunset in La Jolla talking with my son GPS location La Jolla, California Visual analysis A sunset over the ocean in La Jolla Sound analysis Talking with my son at sunset in La Jolla
- 20. 20 A true personal assistant is responsive and proactive “Remember the time I was strolling with my son after the party at La Jolla beach?” “Yes I do, here is a picture you took of the sunset. Should I share it with your family group on WeChat?” Responsive Decision-making and conversation based on contextual analysis and prompting (e.g. finding memories) Proactive Decision-making and conversation based on contextual analysis without prompting (e.g. automatically sharing memories) “I noticed that you are tired and stressed, I’m turning on the Rocky III soundtrack and navigating you to the gym for a workout and sauna.” “This music gets my blood going and a workout and sauna will help me relieve stress.”
- 21. 21 Multi-mic echo cancellation, beamforming, and speech denoising The first step to an on-device virtual assistant Enabling on-device voice UI News SMS Music Maps Wikipedia Weather Stocks Voice activation Service manager Automatic speech recognition (ASR) Natural language understanding (NLU) On-device processing Cloud processing Text-to-speech (TTS)
- 22. 22 Adding an “AI agent” to create a true virtual assistant The on-device AI agent continuously learns personal knowledge and acts intuitively On-device processing Cloud processing (services) Cloud knowledge graph Automatic speech recognition (ASR) Multi-mic echo cancellation and beamforming Sensors Text-to-speech (TTS) Natural language understanding (NLU) Voice activation AI agent
- 23. 23 Adding an “AI agent” to create a true virtual assistant Contextualization allows personalization at acoustic, intent, and behavior levels Cloud processing (services) Automatic speech recognition (ASR) Voice activation Multi-mic echo cancellation and beamforming Cloud knowledge graph On-device processing Text-to-speech (TTS) Sensors Natural language understanding (NLU) AI agent Dialog management Speaker identification Acoustic event detection Gender and age detection Voice activity detection Emotion classification Contextual fusion and learning Local knowledge graph
- 24. 2424 We are advancing AI research to make on-device AI ubiquitous We are creating AI platform innovations that are fundamental to scaling AI across the industry We provide the low-power end-to-end on-device solution for a true personal assistant
- 25. Follow us on: For more information, visit us at: www.qualcomm.com & www.qualcomm.com/blog Thank you! Nothing in these materials is an offer to sell any of the components or devices referenced herein. ©2018 Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved. Qualcomm and Snapdragon are trademarks of Qualcomm Incorporated, registered in the United States and other countries. Other products and brand names may be trademarks or registered trademarks of their respective owners. References in this presentation to “Qualcomm” may mean Qualcomm Incorporated, Qualcomm Technologies, Inc., and/or other subsidiaries or business units within the Qualcomm corporate structure, as applicable. Qualcomm Incorporated includes Qualcomm’s licensing business, QTL, and the vast majority of its patent portfolio. Qualcomm Technologies, Inc., a wholly-owned subsidiary of Qualcomm Incorporated, operates, along with its subsidiaries, substantially all of Qualcomm’s engineering, research and development functions, and substantially all of its product and services businesses, including its semiconductor business, QCT.