An Intro to Text Analytics on Big Data with a use case
1 like•842 views
The document is an introduction to text analytics on big data, highlighting the vast volume of tweets and the potential of extracting structured information from unstructured data. It discusses extraction techniques, including dictionaries and regular expressions, as well as the use of Annotation Query Language (AQL) for developing and refining text extraction features. The content emphasizes the importance of text analytics in handling large data sets and troubleshooting performance issues.
An Intro to Text Analytics on Big Data with a use case
- 1. #TOSMAC Toronto SMAC Meetup – Welcome! An Intro to Text Analytics on Big Data with a use case
- 2. #TOSMAC Toronto SMAC Team | © 2014 IBM Corporation2 Lucas Silva Felipe MosquettaMarcos de Mello
- 3. #TOSMAC Twitters numbers An Intro to Text Analytics on Big Data with a use case | © 2014 IBM Corporation3 As you know: -500 million Tweets are sent per day. -Twitter supports 35+ languages. -255 million monthly active users. Huge amount of data!
- 4. #TOSMAC An Intro to Text Analytics on Big Data with a use case | © 2014 IBM Corporation4 Overview Section1 Section2 Section3 Section4 Section5
- 5. #TOSMAC Section1 Section2 Section3 Section4 Section5 An Intro to Text Analytics on Big Data with a use case | © 2014 IBM Corporation5 Overview
- 6. #TOSMAC Section1 Section2 Section3 Section4 Section5 An Intro to Text Analytics on Big Data with a use case | © 2014 IBM Corporation6 Overview
- 7. #TOSMAC Let’s get started! | © 2014 IBM Corporation7
- 8. #TOSMAC Input data An Intro to Text Analytics on Big Data with a use case | © 2014 IBM Corporation8
- 9. #TOSMAC An Intro to Text Analytics on Big Data with a use case | © 2014 IBM Corporation9 Section2
- 10. #TOSMAC Demo | © 2014 IBM Corporation10
- 11. #TOSMAC Section1 Section2 Section3 Section4 Section5 An Intro to Text Analytics on Big Data with a use case | © 2014 IBM Corporation11 Next section
- 12. #TOSMAC Section1 Section2 Section3 Section4 Section5 An Intro to Text Analytics on Big Data with a use case | © 2014 IBM Corporation12 Next section Extractor: used to extract structured information from unstructured and semi-structured data. AQL: Annotation Query Language. Rule language with familiar SQL-like syntax.
- 13. #TOSMAC Section1 Section2 Section3 Section4 Section5 An Intro to Text Analytics on Big Data with a use case | © 2014 IBM Corporation13 Next section Profiler: troubleshooting performance problems.
- 14. #TOSMAC Main concepts An Intro to Text Analytics on Big Data with a use case | © 2014 IBM Corporation14 Types of extraction specifications: - Dictionaries - Regular expressions - Part of speech
- 15. #TOSMAC Main concepts An Intro to Text Analytics on Big Data with a use case | © 2014 IBM Corporation15
- 16. #TOSMAC Main concepts An Intro to Text Analytics on Big Data with a use case | © 2014 IBM Corporation16
- 17. #TOSMAC An Intro to Text Analytics on Big Data with a use case | © 2014 IBM Corporation17
- 18. #TOSMAC Main concepts An Intro to Text Analytics on Big Data with a use case | © 2014 IBM Corporation18
- 19. #TOSMAC Main concepts An Intro to Text Analytics on Big Data with a use case | © 2014 IBM Corporation19
- 20. #TOSMAC Main concepts An Intro to Text Analytics on Big Data with a use case | © 2014 IBM Corporation20 Types of extraction specifications: - Dictionaries -Regular expressions - Part of speech numbers: 7.5 4 13
- 21. #TOSMAC Demo | © 2014 IBM Corporation21
- 22. #TOSMAC Main concepts An Intro to Text Analytics on Big Data with a use case | © 2014 IBM Corporation22 Types of extraction specifications: - Dictionaries - Regular expressions - Part of speech
- 23. #TOSMAC Main concepts An Intro to Text Analytics on Big Data with a use case | © 2014 IBM Corporation23
- 24. #TOSMAC An Intro to Text Analytics on Big Data with a use case | © 2014 IBM Corporation24
- 25. #TOSMAC | © 2014 IBM Corporation25 An Intro to Text Analytics on Big Data with a use case AQL Guidelines Basic feature AQL statements - Develop the core building blocks of the extractor.
- 26. #TOSMAC | © 2014 IBM Corporation26 An Intro to Text Analytics on Big Data with a use case AQL Guidelines Candidate generation AQL statements - Combine basic features AQL statements.
- 27. #TOSMAC | © 2014 IBM Corporation27 An Intro to Text Analytics on Big Data with a use case Candidate generation AQL statements $7.5 million $4 thousand $ 7.5 million
- 28. #TOSMAC | © 2014 IBM Corporation28 An Intro to Text Analytics on Big Data with a use case Candidate generation AQL statements $7.5 million $4 thousand $ 7.5 million $7.5 million
- 29. #TOSMAC | © 2014 IBM Corporation29 An Intro to Text Analytics on Big Data with a use case AQL Guidelines Filter and consolidate AQL statements - Refine results - Remove invalid annotations - Resolve overlap between annotations.
- 30. #TOSMAC Demo | © 2014 IBM Corporation30
- 31. #TOSMAC | © 2014 IBM Corporation31 An Intro to Text Analytics on Big Data with a use case Conclusion
- 32. #TOSMAC Check point An Intro to Text Analytics on Big Data with a use case | © 2014 IBM Corporation32
- 33. #TOSMAC What we have done An Intro to Text Analytics on Big Data with a use case | © 2014 IBM Corporation33 Section1 Section2 Section3
- 34. #TOSMAC What are we going to do? An Intro to Text Analytics on Big Data with a use case | © 2014 IBM Corporation34 Section4 Section5
- 35. #TOSMAC Demo | © 2014 IBM Corporation35
- 36. #TOSMAC Also using R An Intro to Text Analytics on Big Data with a use case | © 2014 IBM Corporation36 1.75 0.32
- 37. #TOSMAC What are we going to do? An Intro to Text Analytics on Big Data with a use case | © 2014 IBM Corporation37
- 38. #TOSMAC Demo | © 2014 IBM Corporation38
- 39. #TOSMAC So what? An Intro to Text Analytics on Big Data with a use case | © 2014 IBM Corporation39
- 40. #TOSMAC Companies An Intro to Text Analytics on Big Data with a use case | © 2014 IBM Corporation40
- 41. #TOSMAC Exporting to you An Intro to Text Analytics on Big Data with a use case | © 2014 IBM Corporation41
- 42. #TOSMAC Thank you! Let's network! | © 2014 IBM Corporation42