Hadoop by sunitha
Download as PPT, PDF2 likes623 views
This document discusses the growth of data and challenges in storing and analyzing large datasets. It introduces Hadoop as a solution for processing large datasets in parallel across commodity servers. Key aspects of Hadoop covered include its core components HDFS for storage and MapReduce for distributed processing. Example uses by large companies like Amazon and Facebook are listed. The document contrasts Hadoop with RDBMS and explains when Hadoop is preferable to use.
![Hadoop History
• Hadoop was created by Doug Cutting, who named it
after his son's toy elephant .
• 2002-2004 Nutch Open Source web-scale, crawler-
based search
• 2004-2006 Google File System & MapReduce papers
published.Added DFS & MapReduce impl to Nutch
• 2006-2008 Yahoo hired Doug Cutting
• On February 19, 2008, Yahoo! Inc. launched what it
claimed was the world's largest Hadoop production
application
• The Yahoo! Search Webmap is a Hadoop application
that runs on more than 10,000 core Linux cluster and
produces data that is now used in every Yahoo! Web
search query.[22]](https://image.slidesharecdn.com/hadoopbysunitha-120826205259-phpapp01/85/Hadoop-by-sunitha-7-320.jpg)
Hadoop by sunitha
- 2. Data…Data….Data…. • We live in a data world ???? • Total FaceBook Users:835,525,280 (March 31 st 2012) • The New York Stock Exchange generates about one terabyte of new trade data per • day. • • Facebook hosts approximately 10 billion photos, taking up one petabyte of storage http://www.internetworldstats.com/facebook.htm
- 3. Data…is growing ???? From Gantz et al., “The Diverse and Exploding Digital Universe,” March 2008 (http://www.emc.com/ collateral/analyst-reports/diverse-exploding-digital-universe.pdf).
- 4. Problem?? • How do we store and analyze the date??? • one terabyte drives the transfer speed is around 100 MB/s- more than two and a half hours to read all the data off the disk. Writing more slower • We had 100 drives holding one hundredth of the data. • Reliability issues ( failure in hard drive) • Combine data from 100 drives?. • Existing Tools inadequate to process large data sets
- 5. Why can’t we use RDBMS? • An RDBMS is good for point queries or updates, where the dataset has been indexed to deliver low-latency retrieval and update times of a relatively small amount of data. Longer time to read data CPU Memory Disk
- 6. Hadoop is the answer!!!!! • Hadoop is an open source project licensed under the Apache v2 license http://hadoop.apache.org/ • Used for processing large datasets in parallel with the use of low level commodity machines. • Hadoop is build on two main parts. An special file system called Hadoop Distributed File System (HDFS) and the Map Reduce Framework.
- 7. Hadoop History • Hadoop was created by Doug Cutting, who named it after his son's toy elephant . • 2002-2004 Nutch Open Source web-scale, crawler- based search • 2004-2006 Google File System & MapReduce papers published.Added DFS & MapReduce impl to Nutch • 2006-2008 Yahoo hired Doug Cutting • On February 19, 2008, Yahoo! Inc. launched what it claimed was the world's largest Hadoop production application • The Yahoo! Search Webmap is a Hadoop application that runs on more than 10,000 core Linux cluster and produces data that is now used in every Yahoo! Web search query.[22]
- 8. Who uses Hadoop ? Amazon American Airlines AOL Apple eBay Federal Reserve Board of Governors foursquare Fox Interactive Media FaceBook StumbleUpon Gemvara Hewlett-Packard IBM MicroSoft Twitter NYTimes NetFlix Linkedin
- 9. Why Hadoop? • Reliable: The software is fault tolerant, it expects and handles hardware and software failures • Scalable: Designed for massive scale of processors, memory, and local attached storage • Distributed: Handles replication. Offers massively parallel programming model, MapReduce
- 10. What is MapReduce??? – Programming model used by Google – A combination of the Map and Reduce models with an associated implementation – Used for processing and generating large data sets
- 11. MapReduce Explained • The basic idea is that you divide the job into two parts: a Map, and a Reduce. • Map basically takes the problem, splits it into sub-parts, and sends the sub-parts to different machines – so all the pieces run at the same time. • Reduce takes the results from the sub-parts and combines them back together to get a single answer.
- 12. Distributed Grep Split data grep matches Split data grep matches Very All big Split data grep matches cat matches data Split data grep matches
- 14. How Map and Reduce Work Together
- 15. Map Reduce R M E Very Partitioning A D Result big Function P U data C E • Map: – Accepts input key/value pair Reduce : – Emits intermediate key/value pair Accepts intermediate key/value* pair Emits output key/value pair
- 17. RDBMS compared to MapReduce Data Gigabytes Petabytes Size Access Interactive and Batch batch Updates Read and write Write once, read many many times times integrity High Low Scaling Nonlinear Linear Structur Static schema Dynamic schema e
- 18. Hadoop Family Pig A platform for manipulating large data sets Scripting Machine Mahout Machine Learning Algorithms Learning Bigtable-like structured storage HBASE for Hadoop HDFS Non-Rel RDBMS HIVE data warehouse system Non-Rel RDBMS Distribute and replicated data HDFS among machines Hadoop common MapReduce Distribute and monitor tasks Zoo Keeper Distributed Contributed Service
- 19. When to use Hadoop? • Complex information processing is needed • Unstructured data needs to be turned into structured data • Queries can’t be reasonably expressed using SQL • Heavily recursive algorithms • Complex but parallelizable algorithms needed, such as geo-spatial analysis or genome sequencing • Machine learning • Data sets are too large to fit into database RAM, discs, or require too many cores (10’s of TB up to PB) • Data value does not justify expense of constant real-time availability, such as archives or special interest info, which can be moved to Hadoop and remain available at lower cost • Results are not needed in real time • Fault tolerance is critical • Significant custom coding would be required to handle job scheduling • Reference:http://timoelliott.com/blog/2011/09/hadoop-big-data-and- enterprise-business-intelligence.html
- 20. Building Blocks of Hadoop • Running a set of daemons on different servers on the network •NameNode •DataNode •Secondary NameNode •JobTracker •TaskTracker
- 21. • Questions????
- 22. References • Hadoop in Action By Chuck Lam • Hadoop The Definitive Guide By Tom White • http://hadoop.apache.org/