Downloaded 158 times
Uploaded byData Science Thailand
Using hadoop for big data
This document provides an agenda for a workshop on Hadoop and Spark. It begins with background on big data, analytics, and data science. It then outlines workshops that will be conducted on installing and using Hadoop and Spark for tasks like word counting. Real-world use cases for Hadoop are also discussed. The document concludes by discussing trends in Hadoop and Spark.
More Related Content
Similar to Using hadoop for big data
Using hadoop for big data
- 1. Hadoop for (Young)Data Scientist Komes Chandavimol and Team Data Science Lab, Thailand [email protected]
- 2. Agenda • Big Data,Analytics and Data Science • Hadoop + Sparks Workshops • Sharing Experience: Hadoop (Real) Use Cases • Hadoop + Spark Trends,
- 3. 3 Big Data, Analyticsand Data Science
- 4.
- 5.
- 6.
- 7.
- 8.
- 9.
- 11.
- 12.
- 13.
- 14.
- 15. How the DataLake works? 15 http://www.clearpeaks.com/blog/category/tableau Traditional Enterprise Data warehouse
- 16. 16 What you consumefrom Data Lake? https://www.digitalnewsasia.com/business/forget-data-warehousing-its-data-lakes-now
- 17.
- 18.
- 19. 19 Big Data +Analytics = Values https://www.digitalnewsasia.com/business/forget-data-warehousing-its-data-lakes-now
- 20.
- 21.
- 22.
- 23. 23 How to doBig Data Analytics? https://www.digitalnewsasia.com/business/forget-data-warehousing-its-data-lakes-now
- 25. Data Science ExperienceSharing, Big Data Challenge #2,Bangkok Thailand http://drewconway.com/zia/2013/3/26/the-data-science-venn-diagram What is Data Science?
- 26. The Rise ofData Scientist 27 http://flowingdata.com/2009/06/04/rise-of-the-data-scientist/ 2009 https://hbr.org/
- 27.
- 28. Data Science ExperienceSharing, Big Data Challenge #2,Bangkok Thailand http://www.anlytcs.com/2014/01/data-science-venn-diagram-v20.html 2014 The Data Science
- 29. 30 The Solution, DataScience Team
- 30. 31 Data Science Team DoingData Science by O'Neil et al (2013)
- 31. 32 Doing Data Scienceby O'Neil et al (2013)
- 32. 33 Doing Data Scienceby O'Neil et al (2013) Data Science Team Analyzing the Analyzers, Harris (2013)
- 33. 34 Data Science Team DataScientist & Data Engineer http://www.kdnuggets.com/2015/11/different-data-science-roles-industry.html
- 34. 35 Data Science Team DataScientist & Data Engineer http://www.kdnuggets.com/2015/11/different-data-science-roles-industry.html https://www.facebook.com/DataScienceTh/posts/931828353527079:0
- 35.
- 36. 37 Data Science forDummies Pierson (2015) ∗Build In-house Team • Train existing employee • Train existing employee and hire experts • Hire experts ∗Outsourcing requirements to private DS consultants • Outsourcing for comprehensive DS Strategy development • Outsource for DS Solutions to specific problem ∗Leverage Cloud-based platform solutions How to build DS Team?
- 37. Machine Learning Improving Performancein some Task with Experience”. Tom Mitchell Tom Mitchell (1998) The field of study that gives computers the ability to learn without being explicitly programmed. Arthur Samuel (1990) Wikipedia, Data Visualization for Dummies (2014) Data Points: Visualization That Means Something(2013) 38 Machine Learning deals with systems that can learn from data.
- 38.
- 39. Machine Learning Discovery • Class Discovery • Correlation Discovery • Novelty (Surprise) Discovery • Association (or Link Discovery) 40 KirkBorne-workshop-ODSC2016.pdf
- 40. The XYZ ofData Science Smart X : • Smart Cities • Smart Highways • Smart Supply Chain Precision Y : • Precision Medicine • Precision Farming • Precision Pricing Personalized Z : • Personalized Health • Personalized Learning • Personalized Shopping Experience 41 KirkBorne-Workshop-ODSC2016.pdf Intelligence at the edge of the network… at the point of data collection
- 41. 42DataInquest – PredictiveAnalytics and Data Science Bootcamp
- 42. Data Science isa Team Sport http://www.ibmbigdatahub.com/blog/why-data-science-team-sport
- 43.
- 44. 45 Hadoop + SparkWorkshops
- 48. 49 Workshop #1 การติดตั้งHDFS และ YARN
- 50.
- 52. 53 Workshop #3 WordCount(Streaming)
- 53.
- 55. 56 Workshop #5 SetupCloudera QuickStart
- 57. 58 Workshop #6 Exploring HBASEdata in HUE
- 58. 59 Workshop #7 Design aSchema for quick twitter relationship lookup
- 59. 60 Workshop #8 Design aschema for IoT log (Smart Meter)
- 60. 61 Workshop #9 Create anHBase table for Smart meter data
- 61. 62 Workshop #10 Bank CustomerSnapshot
- 64. 65 Workshop #10.1 - 10.1Create Hive Tables 10.2 Create External Hive Tables 10.3 Create External Hive Tables 10.4 Partition
- 66.
- 72.
- 75.
- 83.
- 84. Source: Analytics: TheNew Path to Value, a joint MIT Sloan Management Review and IBM Institute for Business Value study. Copyright © Massachusetts Institute of Technology 2010. Top Performers Use Analytics 5 Times More Than Lower Performers
- 85. Revenue - Cost= Profit
- 86. Monitoring and Maintenance Datasources: IoT Sensors in factory Data products: predictive maintenance models http://www.electrex.it/en/news/600-automated-energy-management-system-a-enms-for-cement-production-plants.ht
- 87. Customer Engagement +Location Data sources: Mobile App, Loyalty Program, GIS Data products: Buying behavior analysis, coupon-response model , location visualization http://www.fastcompany.com/3020859/most-creative-people/how-chinas-one-child-policy-forced-starbucks-to-rethink-its-beijing-sto
- 88. Fuel Saving Data sources:Telematics (sensor), GPS Data products: Prescriptive analytics – route optimization, predictive maintenance (parts/malfunction) http://www.cnet.com/news/ups-turns-data-analysis-into-big-savings/
- 89. Fraud Detection Data sources:historical pattern of transaction data Data products: predictive models – fraud/non-fraudhttps://bluefishway.com/2013/09/13/panic-oh-no-not-again/
- 90. HR Analytics –Google Hiring Data sources: Historical hiring attributes Data products: Predictive model – recruiting high performer Behavioral Test Situational Test GPA Brain Teaser Good School
- 91. Average ROI ofAnalytics/Data Science
- 92. 93 Hadoop + SparkTrends