Presentation on Hadoop Technology
- 2. Index Introduction and History Use and Advantages Issues and Need of Hadoop Users of Hadoop Framework and Architecture HDFS Basic Concept Map Reduce Summery
- 3. Introduction and History • Apache Software Foundation Project • Open Source - Reliable, Scalable, Distributed Computing and Data Storage • Concept: Moving computation > Moving large data. History: • Google File System paper – Oct’2003 • MapReducing & Clustering • Doug Cutting and Mike Carafella in 2005. • Name: Doug Cutting – Yahoo – Feb 2006 • Name Comes - Doug Cutting’s Son (Tohvelant)
- 4. Use & Advantages • Data-intensive text processing • Assembly of large genomes • Graph mining • Machine learning and data mining • Large scale social network analysis Advantages: • Massive Scalability • Flexible Schema • Quicker/Cheaper to set up • Consistence with High Performance • Limitation: Gaps in Analytic Functionality Multiple copies of already big data Inefficient execution & Challenging framework
- 5. Issues and Need Of Hadoop 500 TB per day Over 170 PB Over 6 PB Getting the data to the processors becomes the bottleneck
- 7. Users Of Hadoop And Many More…
- 9. Architecture of Hadoop Master node (single node) Many slave nodes
- 10. HDFS Basic Concept • HDFS works best with a smaller number of large files o Millions as opposed to billions of files o Typically 100MB or more per file • Files in HDFS are write once • Optimized for streaming reads of large files and not random reads
- 11. MapReduce Component • JobTracker & TaskTracker • JobTracker splits up data into smaller tasks(“Map”) and sends it to the TaskTracker process in each node • TaskTracker reports back to the JobTracker node and reports on job progress, sends data (“Reduce”) or requests new jobs
- 13. Summery Open Source Data Management with Scale out Storage High Performance while handling large and Complex data Optimizing for Streaming & Distributed Processing
- 14. Thank You !!