B.MONICA II M.SC COMPUTER SCIENCE
- 1. HADOOP FOUNDATION FOR ANALYTICS BY B.MONICA II M.SC COMPUTER SCIENCE BON SECOURS COLLEGE FOR WOMEN 1
- 2. HADOOP It is an open-source software framework licensed under the Apache v2 license It includes: – Map Reduce : offline computing engine – HDFS : Hadoop distributed file system EXAMPLE 2
- 3. HADOOP GOALS Scalable: It can reliably store and process petabytes. Economical: It distributes the data Efficient: it can process it in parallel on the nodes where the data is located. Reliable: It automatically maintains multiple copies of data 3
- 4. USES FOR HADOOP Data-intensive text processing Assembly of large genomes Graph mining Machine learning and data mining Large scale social network analysis 4
- 5. HADOOP: ASSUMPTIONS Hardware will fail. Applications need a write-once-read-many access model. EXAMPLE Facebook: - To store copies of internal log and dimension data sources - it as a source for reporting/analytics and machine learning - 320 machine cluster with 2,560 cores and about 1.3 PB raw storage 5
- 6. HADOOP CONFIGURATION Conf /hdfs-site.xml: <configuration> <property> <name> Dfs . replication </name> <value> 1 </value> </property> </configuration> 6
- 7. HISTORY OF HADOOP Hadoop was started by Doug Cutting to support two of his other well known projects, Lucene and Nutch Hadoop has been inspired by Google's File System (GFS) which was detailed in a paper by released by Google in 2003 Hadoop, originally called Nutch Distributed File System (NDFS) split from Nutch in 2006 to become a sub-project of Lucene. At this point it was renamed to Hadoop. 7
- 8. EXAMPLE Google search engine 2013 - Hadoop 1.1.2 and Hadoop 2.0.3 alpha. - Ambari , Cassandra, Mahout have been added 8
- 9. • Hadoop is in use at most organizations that handle big data: o Yahoo! o Facebook o Amazon o Netflix 9
- 10. APACHE MAP REDUCE A software framework for distributed processing of large data sets The framework takes care of scheduling tasks, monitoring them and re-executing any failed tasks. It splits the input data set into independent chunks. Map Reduce framework sorts the outputs of the maps, which are then input to the reduce tasks.. 10
- 11. 11
- 12. MAP REDUCE DATAFLOW An input reader A Map function A partition function A compare function A Reduce function An output writer EXAMPLE: JOB TRACKER TASK TRACKER 12
- 13. MAP REDUCE-FAULT TOLERANCE Worker failure: The master pings every worker periodically. Master Failure: It is easy to make the master write periodic checkpoints of the master data structures 13
- 14. JOB TRACKER Tracking Map Reduce jobs in Hadoop Job Tracker performs following actions in Hadoop It accepts the Map Reduce Jobs from client applications Talks to Name Node to determine data location Locates available Task Tracker Node Submits the work to the chosen Task Tracker Node 14
- 15. OTHER TOOLS Hive Hadoop processing with SQL Pig Hadoop processing with scripting Cascading Pipe and Filter processing model H Base Database model built on top of Hadoop Flume Designed for large scale data movement 15
- 16. THANK YOU 16