Watson API Use Case Demos for the Nittany Watson Challenge
1 like1,320 views
1) The document discusses various Watson capabilities including microservices for language, speech, vision, and data, as well as embodied cognition. It provides examples of use cases demonstrating speech to text with multiple speakers, a school navigator chatbot, expertise finder, and multimedia enrichment. 2) Live demos are shown for speech to text with diarization, speech to speech translation, a school finder application, and multimedia processing of video and audio. 3) The multimedia enrichment pipeline is described in detail, outlining how video and audio inputs are processed using various Watson APIs to extract metadata like transcripts, entities, keywords, and visual recognition results.
Watson API Use Case Demos for the Nittany Watson Challenge
- 1. Watson Use Case Demos for the Nittany Watson Challenge January 2017 Mike Pointer, Watson Sr. Solution Architect
- 3. Discovery 25+ WATSON MICROSERVICES 3 Language Services AlchemyLanguage Conversation Document Conversion Language Translator Natural Language Classifier Personality Insights Retrieve and Rank Tone Analyzer Entity Extraction Sentiment Analysis Emotion Analysis Keyword Extraction Concept Tagging Relation Extraction Taxonomy Classification Author Extraction Language Detection Text Extraction Microformats Parsing Feed Detection Linked Data Support Speech Services Speech to Text Text to Speech Vision Services Visual Recognition Similarity Search Data Insight Services AlchemyData News Discovery Tradeoff Analytics Embodied Cognition Services Intu Watson’s APIs are the cognitive building blocks that harness our data. Vision Recognition Conversation
- 4. Use Cases and Demos Speech to Text – Multiple Speakers Chatbot – School Navigator Watson Conversation Service Watson Discovery Service Watson Knowledge Studio
- 6. SPEECH TO TEXT WITH DIARIZATION – MULTIPLE SPEAKERS 6© Copyright IBM Corporation 2016 https://speech-to-text-demo.mybluemix.net/
- 7. Speech to Speech Speech to Text Language Translation Text to Speech 7
- 8. SPEECH TO SPEECH – WITH LANGUAGE TRANSLATION 8© Copyright IBM Corporation 2016 https://speech-to-speech- app.mybluemix.net/?cm_mc_uid=75 059567007614843110528&cm_mc_ sid_50200000=
- 9. Watson Virtual Agent Engagement Your Customers, Students, Faculty, Citizens 9
- 10. © Copyright IBM Corporation 2016 10
- 11. NYC School Finder Matching Personality with Assisted Decision Making (Tradeoff Analytics) 11
- 13. School Navigator Watson Conversation Service Chatbot 13
- 14. ©IBM 2016 Choosing a graduate school is BIG decision, let Watson help… 14 Many students are overwhelmed when deciding which graduate school to attend. Many do not know who to ask, and turn to costly advisors who do not know what is best for their specific needs and goals. THE OPPORTUNITY: Watson can help aspiring graduate students find the best graduate school for them by understanding their preferences, and previous academic scores to build a candidate profile to match to relevant schools. Watson will also educate the user on everything from the admissions process to the best practices for the GMAT exam. The School Navigator uses an interactive interface to engage and advise aspiring students in one of their most important decisions.
- 15. ©IBM 2016 Link: http://schoolnavigator.mybluemix.net/#/ username: watson password: p@ssw0rd SchoolNavigator Demo 15
- 17. EXPERTISE FINDER 17 Challenge Organizations often staff projects with available resources without certainty that the resource is the best fit. That’s because it can be extremely time consuming to align relevant personnel and expertise to projects and tasks. This can lead to under- utilization of resources, delivery challenges, cost overruns and missed opportunities for organic growth. Especially relevant for Financial Services, Law, Research & Development, Consulting & Engineering organizations. Watson inspired solution to the problem Watson can enable enterprises to efficiently locate and identify expertise across the firm. By matching needed expertise with relevant employee experience, Watson streamlines a company’s internal discovery process for resource matching to projects. Expertise Finder - Legal Username: watson Password: w@ts0n
- 18. Multimedia Enrichment Getting concepts from video and audio 18
- 19. ©IBM 2015 IBM Confidential IBM Internal ONLY Live Demo 1/24/2017 19 http://cnn-media.mybluemix.net/#/ Username – cnn Password – cnn123
- 20. ©IBM 2015 IBM Confidential IBM Internal ONLY The Accelerator provides processing of multimedia content to acquire, ingest and enrich it • Input can be any type of multimedia content (images, social media text, video transcripts, audio etc.) and Output is the enriched metadata based on processing them using relevant Watson API’s. • Leverages relevant Watson APIs • Index the enriched content in an efficient storage based on an canoncial schema. 1/24/2017 20
- 21. ©IBM 2015 IBM Confidential IBM Internal ONLY Multimedia Enrichment Pipeline for Audio enrichment 1/24/2017 21 Audio Ingestion (optional) • Extract Audio from the videos • Preserve metadata Transcribe Audio (optional) • Transcribe Audio • Maintain start & end times • Capture word alternatives • Speech to Text Entity Extraction • Invoke AlchemyLanguage and/or Custom WKS AlchemyLanguage Model Keywords • Highly relevant terms & phrases • AlchemyLanguage Expand Keywords • Word2Vec as a API • Trained on Wikipedia and other generally available corpus Tone Analysis • Extract the top emotion, social tone and writing tone • Tone Analysis Sources Pull / Poll Taxonomy & Relationships • Classify or categorize content • Different types of relations between detected entities • AlchemyLanguage Visual Recognition • Use random frames / samples • Class & Face Detection • Visual Recognition Store in Enrichment DB’s (Cloudant, IBM Graph) Closed Caption Transcripts
- 22. ©IBM 2015 IBM Confidential IBM Internal ONLY Multimedia Enrichment Pipeline for Video files 1/24/2017 22 Enrichment Pipeline Service • Pass through all text related pipelines Store in Enrichment DB’s (Cloudant, IBM Graph) VideoProcessorService • Transcribe • ImageCapture/Visual Recognition Video Workload Manager • UI to select file/URI and have ‘processed’, indicate progress, See Results Video URI Status/Complete events Start Event w/ json ID in DB. Insert JSON/Update JSON Read JSON/Update JSON A workload manager would drive workload. We can horizontally scale the VideoProcessor and Enrichment Pipeline. VideoProcessorService • Transcribe • ImageCapture/Visual Recognition Enrichment Pipeline Service • Pass through all text related pipelines
- 23. ©IBM 2015 IBM Confidential IBM Internal ONLY Multimedia Enrichment Pipeline - Video Processor 1/24/2017 23 Video Ingestion • Preserve metadata • Extract Audio from the videos • Generate Sample Frames Transcribe/Normalize Audio • Transcribe Audio • Maintain start & end times • Capture word alternatives • Speech to Text Visual Recognition • Use random frames / samples • Class & Face Detection • Visual Recognition Store in Enrichment DB’s (Cloudant, IBM Graph) json Initial metadata A URI is passed to the Video Ingestion and we use ffmpeg to ingest the video: 1. We instantiate a metadata object and save any existing metadata in the file. 2. a) As the video is read, we pass audio to STT (if we don’t have a transcript) b) Every TIME_INTERVAL ffpmeg saves a Screen capture of the file. 3. We save Transcript, the filename and time taken to the Metadata. 4. Send image URI to VR 5. Update JSON Metadata in DB w/ VR Results. URI PNG image PNG image PNG image PNG image 1 2a 2b 3 4 5 The main idea here is that we must process the file 1 time (at least) and it will take as long as it takes to play the file (for the most part) This process will extract all data possible during the initial processing PRIOR to handing it off to the TranscriptionAnalyzer. One possibility is if we already have a transcription and can skip 2a, we may be able to process video faster for just images.
- 24. ©IBM 2015 IBM Confidential IBM Internal ONLY Multimedia Enrichment Pipeline - Text Enrichment 1/24/2017 24 Entity Extraction • Invoke AlchemyLanguage and/or Custom WKS AlchemyLanguage Model Keywords • Highly relevant terms & phrases • AlchemyLanguage Expand Keywords • Word2Vec as a API • Trained on Wikipedia and other generally available corpus Tone Analysis • Extract the top emotion, social tone and writing tone • Tone Analysis Taxonomy & Relationships • Classify or categorize content • Different types of relations between detected entities • AlchemyLanguage Store in Enrichment DB’s (Cloudant, IBM Graph) Retrieve JSON Document for processing, Update when complete Enrichment Pipeline • Lookup document and retrieve results from Enrichers Each stage will pass on its enriched JSON file to the next stage which will work on the data as it sees fit. At the end of each stage an even to the Enrichment Pipeline indicating the stage is finished will be generated. A set of node.js Streams event: ‘start’ Id: docID event: ‘’finished’ Id: docID
- 26. FACT EXTRACTION WITH WATSON KNOWLEDGE STUDIO 26© Copyright IBM Corporation 2016 http://laser1.watson.ibm.com/sire/ie2.php
- 27. WKS RULES 27