![Example 1 : Color Count
21
Shuffle & Sorting
based on k
Reduce
Reduce
Reduce
Map
Map
Map
Map
Input blocks
on HDFS
Produces (k, v)
( , 1)
Parse-hash
Parse-hash
Parse-hash
Parse-hash
Consumes(k, [v])
( , [1,1,1,1,1,1..])
Produces(kâ, vâ)
( , 100)
Job: Count the number of each color in a data set
Part0003
Part0002
Part0001
Thatâs the output file, it
has 3 parts on probably 3
different machines](https://image.slidesharecdn.com/hadoop-160404132941/85/Hadoop-21-320.jpg)
Hadoop
- 1. Hadoop, a distributed framework for Big Data Presented By: Bhushan Kulkarni T.E(I.T)
- 2. Contents 1. Introduction and Hadoopâs history 2. Architecture in detail 3. Hadoop in industry
- 3. What is Hadoop? âĸ Apache top level project, open-source implementation of frameworks for reliable, scalable, distributed computing and data storage. âĸ It is a flexible and highly-available architecture for large scale computation and data processing on a network of commodity hardware.
- 4. What is Hadoop âĸ Hadoop is a software framework for distributed processing of large datasets across large clusters of computers âĸ Large datasets ī Terabytes or petabytes of data âĸ Large clusters ī hundreds or thousands of nodes âĸ Hadoop is open-source implementation for Google MapReduce âĸ Hadoop is based on a simple programming model called MapReduce âĸ Hadoop is based on a simple data model, any data will fit 4
- 5. Brief History of Hadoop âĸ Google introduced Map reduce Algorithm. âĸ Doug Cutting and team took the solution provided by Google and started an Open Source Project called HADOOP in 2005 and Doug named it after his son's toy elephant.
- 6. Hadoopâs Developers Doug Cutting 2005: Doug Cutting and Michael J. Cafarella developed Hadoop to support distribution for the Nutch search engine project. The project was funded by Yahoo. 2006: Yahoo gave the project to Apache Software Foundation.
- 7. Large-Scale Data Analytics âĸ MapReduce computing paradigm (E.g., Hadoop) vs. Traditional database systems 7 Database vs. ī Many enterprises are turning to Hadoop īĄ Especially applications generating big data īĄ Web applications, social networks, scientific applications
- 8. Why Hadoop is able to compete? 8 Scalability (petabytes of data, thousands of machines) Database vs. Flexibility in accepting all data formats (no schema) Commodity inexpensive hardware Efficient and simple fault- tolerant mechanism Performance (tons of indexing, tuning, data organization tech.) Structured Data
- 9. Key Components âĸ Hadoop framework consists on two main layers âĸ Distributed file system (HDFS) âĸ Execution engine (MapReduce) 9
- 10. Hadoop: How it Works 10
- 11. Hadoop Architecture 11 Master node (single node) Many slave nodes âĸ Distributed file system (HDFS) âĸ Execution engine (MapReduce)
- 12. Hadoop Distributed File System (HDFS) 12 Centralized namenode - Maintains metadata info about files Many datanode (1000s) - Store the actual data - Files are divided into blocks - Each block is replicated N times (Default = 3) File F Blocks (64 MB)
- 13. Main Properties of HDFS âĸ Large: A HDFS instance may consist of thousands of server machines, each storing part of the file systemâs data âĸ Replication: Each data block is replicated many times (default is 3) âĸ Failure: Failure occurs rarely âĸ Fault Tolerance: Detection of faults and quick, automatic recovery from them is a core architectural goal of HDFS âĸ Namenode is consistently checking Datanodes 13
- 14. What is MapReduce? âĸ MapReduce is a programming model âĸ Programs written in this functional style are automatically parallelized and executed on a large cluster of commodity machines âĸ MapReduce is an associated implementation for processing and generating large data sets. MapReduce MAP map function that processes a key/value pair to generate a set of intermediate key/value pairs REDUCE and a reduce function that merges all intermediate values associated with the same intermediate key.
- 15. Properties of MapReduce Engine âĸ Job Tracker is the master node (runs with the namenode) âĸ Receives the userâs job âĸ Decides on how many tasks will run (number of mappers) âĸ Decides on where to run each mapper (concept of locality) 15 âĸ This file has 5 Blocks ī run 5 map tasks âĸ Where to run the task reading block â1â âĸ Try to run it on Node 1 or Node 3 Node 1 Node 2 Node 3
- 16. Properties of MapReduce Engine (Contâd) âĸ Task Tracker is the slave node (runs on each datanode) âĸ Receives the task from Job Tracker âĸ Runs the task until completion (either map or reduce task) âĸ Always in communication with the Job Tracker reporting progress 16
- 17. The Programming Model Of MapReduce Map takes an input pair and produces a set of intermediate key/value pairs. The MapReduce library groups together all intermediate values associated with the same intermediate key. The Reduce function, also written by the user, accepts an intermediate key I and a set of values for that key. It merges together these values to form a possibly smaller set of values
- 18. MapReduce data flow with a single reduce task
- 19. MapReduce data flow with multiple reduce tasks
- 20. MapReduce data flow with no reduce tasks
- 21. Example 1 : Color Count 21 Shuffle & Sorting based on k Reduce Reduce Reduce Map Map Map Map Input blocks on HDFS Produces (k, v) ( , 1) Parse-hash Parse-hash Parse-hash Parse-hash Consumes(k, [v]) ( , [1,1,1,1,1,1..]) Produces(kâ, vâ) ( , 100) Job: Count the number of each color in a data set Part0003 Part0002 Part0001 Thatâs the output file, it has 3 parts on probably 3 different machines
- 22. Example 2: Color Filter 22 Job: Select only the blue and the green colors Input blocks on HDFS Map Map Map Map Produces (k, v) ( , 1) Write to HDFS Write to HDFS Write to HDFS Write to HDFS âĸ Each map task will select only the blue or green colors âĸ No need for reduce phase Part0001 Part0002 Part0003 Part0004 Thatâs the output file, it has 4 parts on probably 4 different machines
- 23. Why use Hadoop? âĸ Need to process Multi Petabyte Datasets âĸ Data may not have strict schema âĸ Expensive to build reliability in each application âĸ Need common infrastructure âĸ Very Large Distributed File System âĸ Assumes Commodity Hardware âĸ Optimized for Batch Processing
- 24. Who Uses MapReduce/Hadoop âĸ Google: Inventors of MapReduce computing paradigm âĸ Yahoo: Developing Hadoop open-source of MapReduce âĸ IBM, Microsoft, Oracle âĸ Facebook, Amazon, AOL, NetFlex âĸ Many others + universities and research labs 24
- 25. THANK YOU!! 25