SlideShare a Scribd company logo

An Introduction to Hadoop

Download as KEY, PDF
Dan Harvey
Dan Harvey

Hadoop is an open-source software framework for distributed storage and processing of large datasets across clusters of computers using simple programming models. It allows users to write programs that process vast amounts of data in parallel by breaking the computations into small sub-problems, each of which can be solved in a separate processing unit and then combining the results. The key aspects are MapReduce, which splits the workload into mapping and reducing functions, and the distributed file system, which stores data across clusters and allows processing to be parallelized.

Technology
1 of 19
Downloaded 34 times
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
An Introduction to Hadoop By Dan Harvey
“The Apache Hadoop project develops open-source software for reliable, scalable, distributed computing”
Into the past • Doug Cutting: Lucene search library • Linear merge sorted indexes • Disk was the bottleneck • How to speed and scale up?
Split over more disks... Then more machines...
Jim Grey on Disks Throughput on SATA disk 49.1 MBps 50 37.5 Throughput in MBps t Random 25 Sequential 12.5 0.6 MBps 0 Access Type Barclay, T., Chong, W., & Gray, J. (2003). A Quick Look at Serial ATA Disk Performance.
Jim Grey on Disks Time to read 2TB disk 50 40.5 days 37.5 Time in Days 0.6 MBps 25 49.1 MBps 12.5 0.5 days 0 Throughput
So... • Use more disks & machines • Use sequential disk access • Linear merge == sequential access! • But how to make it accessible?
MapReduce: Simplified Data Processing on Large Clusters Jeffrey Dean and Sanjay Ghemawat jeff@google.com, sanjay@google.com Google, Inc. Abstract given day, etc. Most such computations are conceptu- ally straightforward. However, the input data is usually MapReduce is a programming model and an associ- large and the computations have to be distributed across ated implementation for processing and generating large hundreds or thousands of machines in order to finish in data sets. Users specify a map function that processes a a reasonable amount of time. The issues of how to par- key/value pair to generate a set of intermediate key/value allelize the computation, distribute the data, and handle pairs, and a reduce function that merges all intermediate failures conspire to obscure the original simple compu- values associated with the same intermediate key. Many tation with large amounts of complex code to deal with real world tasks are expressible in this model, as shown these issues. in the paper. As a reaction to this complexity, we designed a new Programs written in this functional style are automati- abstraction that allows us to express the simple computa- cally parallelized and executed on a large cluster of com- tions we were trying to perform but hides the messy de- modity machines. The run-time system takes care of the tails of parallelization, fault-tolerance, data distribution Google’s MapReduce details of partitioning the input data, scheduling the pro- and load balancing in a library. Our abstraction is in- gram’s execution across a set of machines, handling ma- spired by the map and reduce primitives present in Lisp chine failures, and managing the required inter-machine and many other functional languages. We realized that communication. This allows programmers without any most of our computations involved applying a map op- experience with parallel and distributed systems to eas- eration to each logical “record” in our input in order to ily utilize the resources of a large distributed system. compute a set of intermediate key/value pairs, and then Our implementation of MapReduce runs on a large applying a reduce operation to all the values that shared the same key, in order to combine the derived data ap-
Map, Shuffle, Reduce
Distributed Storage MapReduce: Simplified Data Processing on Large Clusters Jeffrey Dean and Sanjay Ghemawat MapReduce: Simplified Data Processing on Large Clusters jeff@google.com, sanjay@google.com Google, Inc. Jeffrey Dean and Sanjay Ghemawat MapReduce: Simplified Data Processing on Large Clusters jeff@google.com, sanjay@google.com Abstract given day, etc. Most such computations are conceptu- Google, Inc. ally straightforward. However, the input data is usually Jeffrey Dean and Sanjay Ghemawatmodel and an associ- MapReduce is a programming large and the computations have to be distributed across ated implementation for processing and generating large hundreds or thousands of machines in order to finish in jeff@google.com, sanjay@google.com function that processes a data sets. Users specify a map MapReduce: Simplified Data Processing on Large Clusters key/value pair to generate a set of intermediate key/value a reasonable amount of time. The issues of how to par- allelize the computation, distribute the data, and handle Abstract a reduce function that merges all intermediate such computations are conceptu- pairs, Google, Inc. and given day, etc. Most failures conspire to obscure the original simple compu- values associated with the same intermediate key. ManyHowever, the input data is usually MapReduce is a programming model and an associ- ally straightforward. tation with large amounts of complex code to deal with Jeffrey Dean and Sanjay Ghemawat real world tasks are expressible largemodel, computations have to be distributed across this and the ated implementation for processing and generating large in hundreds or as shown of these issues. order to finish in thousands machines in data sets. Users specify a in thefunction that processes a map paper. a reasonable amount of time. Thereaction to thisto par- As a issues of how complexity, we designed a new jeff@google.com, sanjay@google.com Abstract Programs written key/value Most such are automati- areabstraction that allows given day, etc. computations conceptu- key/value pair to generate a set of intermediate in this functional stylethe computation, distribute the data, and us to express the simple computa- allelize handle ally straightforward.large clusterthe com- data is usually However, of input Google, Inc. MapReduce is apairs, and a reduce function that merges all intermediate on a programming model and cally parallelized andand the computations conspirebe distributed across trying to perform but hides the messy de- an associ- large executed failures have to to obscure thewe were simple compu- tions original values associated with the modity machines. The run-time system takes care of the same intermediate key. Many ated implementation for processing and generating large tation with large amounts tails of parallelization, fault-tolerance, data distribution of finish in code to deal with complex hundreds or thousands of machines in order to data sets. Users specifyworld tasks are expressible in this model, as shown data, scheduling the pro- of how to par- real a map function that processes partitioning the input these issues. details of a and load balancing in a library. Our abstraction is in- gram’s execution reasonable amount of time. The issues key/value pair to generatepaper.of intermediate key/value in the a set a across a set of machines, handling ma- As distribute to data, and handle map and reduce primitives present in Lisp spired by the allelize the computation, a reaction thethis complexity, we designed a new chine failures, and managing the required inter-machine pairs, and a reduce function that merges all intermediate style are automati- toabstraction that allowssimple compu-other functional languages. We realized that and manythe simple computa- Abstract given day, etc. Most such computations are conceptu- Programs written in this functional conspire obscure the original us to express failuresallows Replicated Blocks communication. This of com- programmers without any values associated with the same intermediate key. on a large cluster with large amounts of complex code to deal but hides the messy involved applying a map op- cally parallelized and executed Many most of our computations de- ally straightforward. However, the input data is usually tation tions we were trying to perform with MapReduce is a programming model and an associ- large and the computations have to be distributed across real world tasks are expressible in this model, as shown withthese issues. distributedof parallelization, fault-tolerance, data distribution in our input in order to experience parallel and modity machines. The run-time system takes care of the tails systems to eas- eration to each logical “record” ated implementation for processing and generating large in the paper. ily utilize the resources ofpro- a large distributed system. details of partitioning the input data, scheduling a reaction to and load balancing in a library. new set of intermediate key/value pairs, and then compute a hundreds or thousands of machines in order to finish in As the this complexity, we designed a Our abstraction is in- data sets. Users specify a map function that processes a a reasonable amount of time. The issues of how to par- gram’s functional style a setOurmachines, handling ma- allows usrunsthe map and reduce primitives present in Lisp all the values that shared Programs written in thisexecution across are automati-of implementation of that abstraction spired to express the simple computa- reduce operation to MapReduce by on a large applying a key/value pair to generate a set of intermediate key/value allelize the computation, distribute the data, and handle cally parallelized and executed on and managing of com-commodity machines and to many other functional messy de- on Largethat chine failures, a large cluster the requiredtions we were trying is performscalable:Processing We realized combine the derived data ap- cluster of MapReduce: Simplified Data inter-machine and highly but hides thethe same key, in order to Clusters languages. pairs, and a reduce function that merges all intermediate failures conspire to obscure the original simple compu- communication. This allows programmers without any care of the most of our computations propriately. Our use mapaop- modity machines. The run-time system takes a typical MapReduce computation processes many ter- involved applying a of functional model with user- tails of parallelization, fault-tolerance, data distribution values associated with the same intermediate key. Many tation with large amounts of complex code to deal with experience data, scheduling distributed on thousands of machines. Programmers “record” in map and in order to abytes of data details of partitioning the inputwith parallel and the pro- systems to balancingeration to each Our abstraction is in- and load eas- in a library. logical specified our input reduce operations allows us to paral- real world tasks are expressible in this model, as shown these issues. a large system easy to by the map Jeffrey Dean and Sanjay in large computations then and to use re-execution gram’s execution across a set the machines, of find thema- spired use: MapReduce intermediate key/value pairs, and easily ily utilize of resources handling distributed system. hundreds ofreduce set of pro- present Ghemawat compute a primitives and lelize Lisp in the paper. As a reaction to this complexity, we designed a new MapReduce: Simplified Data Processing one thou- as the primary grams have been implemented and applying a on Large Clustersvalues that shared fault tolerance. upwards ofreduce operation to all the mechanism for chine failures, and managing the required inter-machine runs on a large functional languages. We realized that Our implementation of MapReduce and many other Programs written in this functional style are automati- abstraction that allows us to express the simple computa- sand MapReduce jobs are executed onjeff@google.com, sanjay@google.com contributions ap-this work are a simple and cluster of programmers without any Google’s clusters to combine the derived data of the same key, in order communication. This allows commodity machines and is highly of our computations involved applying a map op- The major most scalable: cally parallelized and executed on a large cluster of com- tions we were trying to perform but hides the messy de- experience with parallel andMapReduce systems today.processes many each logical “record” Our use of apowerfulto model that enables automatic parallelization every eas- a typical distributed computation eration to ter- propriately. in our input in order interface with user- functional modity machines. The run-time system takes care of the tails of parallelization, fault-tolerance, data distribution ily utilize the resources of oflarge distributed system. specified mapGoogle, Inc. distribution of large-scale computations, combined and abytes a data on thousands of machines.Dean and set of intermediate key/value pairs, and then allows us to paral- Jeffrey compute a Sanjay Ghemawatand reduce with an implementation of this interface that achieves Programmers operations details of partitioning the input data, scheduling the pro- and load balancing in a library. Our abstraction is in- Our implementation the system easy to use:1hundreds of MapReduceapro- find of MapReduce runs on a large lelize large computations easily and to use re-execution applying reduce operation to all the values that shared Introduction gram’s execution across a set of machines, handling ma- spired by the map and reduce primitives present in Lisp MapReduce: Simplified scalable: the one key, Large Clusters derived Section 2 describeslargebasic programming model and cluster of commodity machines and is highlyData upwards ofsame thou- in order toprimary mechanism for fault tolerance. grams have been implemented and Processing on as the combine the jeff@google.com, sanjay@google.com high performance on data ap- the clusters of commodity PCs. chine failures, and managing the required inter-machine and many other functional languages. We realized that sand MapReduceprocessesexecuted on Google’s clusters useThe a functional model with user- are a simple and a typical MapReduce computation jobs are many ter- propriately. Our of major contributions of this work communication. This allows programmers without any most of our computations involved applying a map op- every day. abytes of data on thousands of machines. Programmers AbstractInc. Google, Over the past five years, mapauthors and many othersthat enables paral- Most such computations are conceptu- specified the and powerful interface allows us to automatic parallelization reduce operations at given day, etc. examples. Section 3 describes an imple- gives several Google have implemented computationsspecial-purpose ally straightforward.MapReduce interface tailored towards lelize large hundreds of easily andof use re-executionof the combined mentation experience with parallel and distributed systems to eas- eration to each logical “record” in our input in order to find the system easy to use: hundreds of MapReduce pro- is a programming model and an associ- large-scale computations,However, the input data is usually and distribution to Jeffrey Dean and Sanjay Ghemawat mechanism for fault tolerance.ourand interface that achieves to be distributed acrossde- MapReduce that process large amounts of raw data, large cluster-based computing environment. Section 4 ily utilize the resources of a large distributed system. compute a set of intermediate key/value pairs, and then computations as the primary with an implementation of this the computations have grams have been implemented and upwards of one thou- Our implementation of MapReduce runs on a large applying a reduce operation to all the values that shared 1 Introduction ated implementationdocuments, contributions of thisetc.,on largescribes and such as crawled for processing and generating large to hundreds several refinements of the programming model jeff@google.com,Users The map web high performance request logs, sand MapReduce jobs are executed on Google’s clusters specify amajor function that processes a are a simple orof commodity machines in order to finish in work clusters thousands of PCs. data sets. sanjay@google.comderived data, such as inverted a reasonable amount ofuseful. The issues of how to par- compute various kinds of cluster of commodity machines and is highly scalable: the same key, in order to combine the derived data ap- every day. Abstract given day, etc. 2 describes parallelization are conceptu- Section 5 has performance Section automatic computations found time. that we have powerful interface that enables Most suchthe basic programming model and key/value pair to generate a set of intermediate key/value a typical MapReduce computation processes many ter- propriately. Our use of a functional model with user- Over the past five years, the authors and many others at allyoflarge-scale computations,measurements of usually indices, various representations gives severalstructure allelize inputdescribes an distribute the data, and variety of the graph examples. Section Google, Inc. summaries of straightforward. However,tasks.the3computation, imple- combineddata is the our implementation for a handle pairs, and documents, distribution of and MapReduce is a programming web a reduce function that merges allthe computations have tointerface tailored towardsoriginalMapReduce within intermediate abytes of data on thousands of machines. Programmers specified map and reduce operations allows us to paral- Google have implementedof model andspecial-purpose largethe number of pages failures beSection to explores the use of simple compu- hundreds of an associ- with an implementation of this the MapReduce conspire obscure and mentation of interface that achieves distributed across the 6 lelize large computations easily and to use re-execution 1 Introduction ated computations that process large amounts of the same most our cluster-based computing environment. Section 4 complex code toitdeal with implementation for processingassociated with raw data,intermediate key. Many of machines in includingfinish in de- values andper host, the set of crawled generating large frequentthousands a tation with order amounts of hundreds or queries in Google large to our experiences in using as the basis find the system easy to use: hundreds of MapReduce pro- high performance on large clusters of commodity PCs. as the primary mechanism for fault tolerance. data such as crawled documents, web tasks arelogs, etc., toin reasonableseveral refinements of issues of how to par- sets. Users specify a map functionrequest expressible a this model,amount of time. The the programming model real world that processes a scribes as shown these issues. grams have been implemented and upwards of one thou- Section 2 describes the basic programming model and key/value pairvarious kinds setthe intermediate key/value to generate a in derived data, such as invertedallelize the computation, useful. Section 5 has this complexity, we designed a new of paper. sand MapReduce jobs are executed on Google’s clusters The major contributions of this work are a simple and Over the past five Abstractauthors and many others at given day,several examples.we have found distribute imple- to performance compute years, the of that Section 3 describes ana the As reaction gives etc. Most such computations are conceptu- data, and handle pairs, and a reduce function that merges written in this functional measurements of our implementation allows variety of the simple computa- Programs all intermediate style are automati- every day. powerful interface that enables automatic parallelization Google have implemented hundredsrepresentations of the graph structure failures conspire on Operating usually that for andus to express of MapReduce abstraction indices, various of special-purpose ally mentation ’04:the However, theinterface tailoredoriginal simple a Implementation straightforward.a large cluster ofto obscureisthe towards Design compu- values associated with and USENIX Association executed on6th Symposium inputexplores Systemsof MapReduce withinbut hides the messy de- 137 the same intermediate and OSDI cally parallelized key. Many MapReduce is athat of web documents, summaries of the number the pages computing to com- data large be distributed thewe 4 de- trying to with tation with Section 6amounts ofSection were to deal perform tions use code complex and distribution of large-scale computations, combined computations programming model an associ- process large amounts of raw data, and of computations have environment. across our cluster-based tasks. with an implementation of this interface that achieves real crawled per and generating in this largeThe run-time ated such as crawled for processing are expressible large model,scribes severalthese issues. including our totails of in in using it as fault-tolerance, data distribution world tasks host, the set of most machines. as shown implementation documents, web request modityetc., frequent queries in asystem takes carethe order experiences Google of in theof parallelization, the basis logs, to hundreds or thousands of machines refinements programming model finish 1 Introduction high performance on large clusters of commodity PCs. data compute various theapaper.derived data, such as inverteda reasonable have data, schedulingSection 5 complexity, we designed aanew in sets. Users specify map function that processesof partitioning the input found time. The the pro- hasand load balancing in library. Our abstraction is in- kinds of details a As a reaction to this that we amount of useful. issues of how to par- performance Section 2 describes the basic programming model and key/value pair to generate a set of intermediatefunctional style are measurementsmachines, handling ma- forspired by the map and reduce primitives present in Lisp indices, various representations of gram’s execution across a set of abstraction that allows us to express the simple computa- Programs written inthe graph structure this key/value automati- allelize the computation, distribute the data, anda handle of of our implementation variety Over the past five years, the authors and many others at gives several examples. Section 3 describes an imple- pairs, and a reduce cally parallelized of the number of large cluster of com- the tions we wereoriginalto perform many other functional languages. We realized that function that merges executed on failures,failures conspire to required inter-machine of web documents, summaries and all intermediate chine a pages and managing and but hides tasks. Section 6 explores thetrying simple compu- obscure the use of MapReduce within the messy de- Google have implemented hundreds of special-purpose mentation of the MapReduce interface tailored towards values associated host, the same intermediate key.’04: 6thin takes This of the programmerscomplex inand to most with data distribution modity set of most frequent queries Symposium allows tails Systems Design using it as of basis USENIX Association OSDI system communication. care on Operating of parallelization, fault-tolerance, without any Implementation crawled per with the machines. The run-timeMany a tation with large amounts experiences code deal the our computations involved applying a map op- Google including our of 137 computations that process large amounts of raw data, our cluster-based computing environment. Section 4 de- real world tasks are expressible in this model, experience with parallelpro- distributed systems to eas-a library. Our each logical is in- details of partitioning the inputshownscheduling the and as data, these issues. a largeand load balancing in eration to abstraction “record” in our input in order to in the paper. gram’s execution across a set of machines, handlingof ily utilize the resources ma- distributed system. compute a set of intermediate key/value pairs, and then such as crawled documents, web request logs, etc., to scribes several refinements of the programming model As a reaction to spired by the mapwe designed primitives present in Lisp this complexity, and reduce a new compute various kinds of derived data, such as inverted that we have found useful. Section 5 has performance chine failures, and managing the required inter-machine Programs written in this functional style are automati- and many other the simple computa-a reduce operation to all the values that shared Our implementation of MapReduce runs on a large applying abstraction that allows us to express functional languages. We realized that communication. This allows programmers without any the same key, 137in order to combine the derived data ap- indices, various representations of the graph structure measurements of our implementation for a variety of cally parallelized and executed’04:a6th Symposium com- commodity machines mostis Implementation the involved applying a map op- USENIX Association OSDI on large cluster of on of cluster Operating Systems Design andofhighly scalable: and tions we were trying to performcomputations messy de- our but hides of web documents, summaries of the number of pages tasks. Section 6 explores the use of MapReduce within experience with parallel and distributed systems computation processes many ter- “record” in our input in orderato modity machines. The run-time system takes care of the MapReduceto eas- a typical eration to each logical propriately. Our use of functional model with user- tails of parallelization, fault-tolerance, data distribution ily utilize the resources of a abytesdistributed thousands of machines. a set of intermediate key/value pairs, reduce operations allows us to paral- details of partitioning the input data, scheduling the pro- dataandsystem. large of on Programmers specified map and and then crawled per host, the set of most frequent queries in a Google including our experiences in using it as the basis load balancingcompute in a library. Our abstraction is in- find ma- applying a primitives present in Lisp computations easily and to use re-execution spired a the map and reduce reduce Our implementation of MapReduce runs on by large pro- lelize large gram’s execution across a set of machines, handlingthe system easy to use: hundreds of MapReduceoperation to all the values that shared as the primary mechanism for fault tolerance. chine failures, and cluster of commodity machines and is been implemented functional languages. Weto combine the derived data ap- managing the required inter-machine highlymany other and upwards of one order realized that grams have and scalable: the same key, in thou- communication. This allows programmers without MapReduce jobs our executed on Google’s clusters a afunctional model with user- this work are a simple and sand any a typical MapReduce computation processes many ter- most of are propriately. Our use of computations involved applying map op- The major contributions of USENIX Association OSDI ’04: 6th Symposium on Operating Systems Design and Implementation 137 every day. specified map and reduce operations interface thatparal- automatic parallelization powerful allows us to enables
Data Locality: Map MapReduce: Simplified Data Processing on Large Clusters Jeffrey Dean and Sanjay Ghemawat MapReduce: Simplified Data Processing on Large Clusters jeff@google.com, sanjay@google.com Google, Inc. MapReduce, Simplified, Data, Processing, Jeffrey Dean and Sanjay Ghemawat MapReduce: Simplified Data Processing on Large Clusters jeff@google.com, sanjay@google.com Large, Clusters, Jeffrey, Dean, and, Sanjay, Google, Inc. Jeffrey Dean and Sanjay Ghemawat jeff@google.com, sanjay@google.com Ghemawat, Google, Inc, ... Abstract Google, Inc. given day, etc. Most such computations are conceptu- ally straightforward. However, the input data is usually MapReduce is a programming model and an associ- large and the computations have to be distributed across ated implementation for processing and generating large hundreds or thousands of machines in order to finish in data sets. Users specify a map function that processes a a reasonable amount of time. The issues of how to par- key/value pair to generate a set of intermediate key/value allelize the computation, distribute the data, and handle pairs, and a reduce function that merges all intermediate Map Stage failures conspire to obscure the original simple compu- values associated with the same intermediate key. Many MapReduce: Simplified Dataof complex code to deal with real world tasks are expressible in this model, as shown tation with large amounts Processing on Large Clusters these issues. in the paper. As a reaction to this complexity, we designed a new Programs written in this functional style are automati- abstraction that allows us to express the simple computa- cally parallelized and executed on a large cluster of com- Jeffrey Dean and Sanjay Ghemawat tions we were trying to perform but hides the messy de- modity machines. The run-time system takes care of the tails of parallelization, fault-tolerance, data distribution jeff@google.com, sanjay@google.com details of partitioning the input data, scheduling the pro- and load balancing in a library. Our abstraction is in- gram’s execution across a set of machines, handling ma- spired by the map and reduce Inc. Google, primitives present in Lisp chine failures, and managing the required inter-machine and many other functional languages. We realized that communication. This allows programmers without any most of our computations involved applying a map op- experience with parallel and distributed systems to eas- eration to each logical “record” in our input in order to ily utilize the resources of a large distributed system. compute a set of intermediate key/value etc. Most then computations are conceptu- Abstract given day, pairs, and such Our implementation of MapReduce runs on a large applying a reduce operation to all straightforward.shared ally the values that However, the input data is usually cluster of commodity machinesMapReduce is a programming model key, an order to combineand the computations have to be distributed across and is highly scalable: the same and in associ- large the derived data ap- ated implementation for processing and generating large a functional model with user-machines in order to finish in a typical MapReduce computation processes many ter- propriately. Our use of hundreds or thousands of abytes of data on thousands of machines. Programmersmap specified that etc. Most a operations allows are to paral- The issues of how to par- Abstract Users specify a data sets. function mapprocesses and reduce us MapReduce, programming, model, associ- given day, such a reasonable amount of time. computations conceptu- find the system easy to use: hundreds of MapReduce pro- set ofally straightforward. However,and to use data is usually key/value pair to generate a lelize large computations easily intermediate key/value allelize input re-execution the the computation, distribute the data, and handle grams have been is a programmingupwardsandone associ-that merges allthe mechanism forhave to be distributed across the original simple compu- MapReduce implemented and and a reduce function aslargeprimary computations failures conspire to obscure pairs, model of an thou- the and intermediate fault tolerance. sand MapReduce jobs for processingassociated with the same intermediatecontributions of tation withare a simple andin complex code to deal with values and generating large ated implementation are executed on Google’s clusters The major key. Many this work large hundreds or thousands of machines in order amounts of to finish ated, implementation, processing, every day. Users specify a map function that are expressiblepowerful interface shown time. automatic parallelization data sets. real world tasks processes a in this model, as that enables a reasonable amount of these issues. of how to par- The issues in the intermediate key/value and distribution of large-scale computations, combined key/value pair to generate a set of paper. Abstract allelize the computation, distribute the data, to this complexity, we designed a new As a reaction and handle pairs, and a reduce function that merges written day, etc. Most suchconspire to obscureabstraction that allows us to express the simple computa- Programs all intermediate with an implementation are this interface that achieves given in this functional stylecomputations of conceptu- failures are automati- 1 Introduction the cally parallelized key. Many highHowever, the on large clustersoriginal simple PCs. ally straightforward. performance input data isthe of commodity compu- usually apReduce is avalues associated with and same intermediate and executed on a large cluster ofamounts of complex code to deal with programming model an associ- tation with large com- tions we were trying to perform but hides the messy de- generating, large, data, sets, Users, ... real world tasks are expressible large model, The run-time system takes carebe distributedprogramming model and mplementation for processing and generating modity this large as shown in machines. and the computations have to of the basic across Section 2 describes the these issues. tails of parallelization, fault-tolerance, data distribution Over the past five years, the authors and many othersor thousands several examples. Section load balancing imple- hundreds at gives of machines in order toand 3 describes an in library. Our abstraction is in- finish in sets. Users specify apaper. function that processesof partitioning the input data, scheduling the pro- complexity, we designed aanew in the map details a As a reactionissues of to this Google have implemented hundreds execution across a set mentation of The MapReduce interface tailored towards gram’s of a reasonable amount of time. that allows us how to par- simple computa- Programs written in this key/value special-purpose the alue pair to generate a set of intermediatefunctional style are automati- of machines, handling ma- to spired by the map and reduce primitives present in Lisp abstraction express the allelize the computation, distributecomputing environment. Section 4 de- the data, and handle computations that process large amounts of raw data, the required inter-machine chine failures, and managing our cluster-based and many other functional languages. We realized that and a reduce function that merges all intermediate such as crawled documents, communication. This conspire scribes several original simple compu- computations involved applying a map op- failures allows to obscure the refinements web request logs, etc., to programmers without any of the of our most programming model s associated with the same intermediate key. Many tation with and amounts of complex eas- to deal with large that we systems to code compute various kinds of derived data, such as inverted distributedhave found useful. Section to has performance eration 5 each logical “record” in our input in order to world tasks are expressible in this model, experience with parallel as shown these issues. indices, various representations utilize the resources of a large distributed system. implementation afor a variety of ily of the graph structure measurements of our compute set of intermediate key/value pairs, and then paper. of web documents, summaries of the number ofapages As reaction to this complexity, we designed a of MapReduce within use new tasks. Section 6 explores the applying a reduce operation to all the values that shared ograms written in thisper host, the set are most frequent abstraction that allows us including athe experiences in using it as the basis Our implementation of MapReduce runs on large crawled functional style of automati- queries in a Google to expressour simple computa- key, in order to combine the derived data ap- cluster of commodity machines and is highly scalable: the same a typical MapReduce computation processes many ter- propriately. Our use of a functional model with user- abytes of data on thousands of machines. Programmers specified map and reduce operations allows us to paral- lelize large computations easily and to use re-execution USENIX Association OSDI ’04:find the system easy to use: hundreds of MapReduce pro- 6th Symposium on Operating Systems Design and Implementation 137 grams have been implemented and upwards of one thou- as the primary mechanism for fault tolerance. sand MapReduce jobs are executed on Google’s clusters The major contributions of this work are a simple and every day. powerful interface that enables automatic parallelization and distribution of large-scale computations, combined with an implementation of this interface that achieves 1 Introduction high performance on large clusters of commodity PCs. Section 2 describes the basic programming model and Over the past five years, the authors and many others at gives several examples. Section 3 describes an imple-
Shuffle and, associated, clusters, data, data, dean, MapReduce, Simplified, Data, Processing, implementation, inc, jeffrey, large, large, Large, Clusters, Jeffrey, Dean, and, Sanjay, mapreduce, mapreduce, model, generating, Ghemawat, Google, Inc, ... ghemawat, google, google, ... MapReduce, programming, model, programming, processing, processing, associated, implementation, processing, sanjay, simplified, sets, users, ... generating, large, data, sets, Users, ...
Reduce and, 1 associated, 1 clusters, 1 and, associated, clusters, data, data, dean, data, 2 implementation, inc, jeffrey, large, large, ... mapreduce, mapreduce, model, generating, mapreduce, 2 ghemawat, google, google, ... model, 1, generating, 1 ghemawat, 1 google, 2 programming, 1 processing, 2 programming, processing, processing, sanjay, 1 sanjay, simplified, sets, users, ... ...
So Hadoop? • Map Reduce for processing • Distributed File System (HDFS) for storage
Other projects
Uses • Log analysis • Data Warehouse • Natural Language Processing • Search indexes • Machine Learning • ...
1.0 and beyond • Next generation Hadoop - YARN • Not just MapReduce • General resource allocation framework
Users
Questions?

More Related Content

Similar to An Introduction to Hadoop (20)

PDF
Data Processing in the Work of NoSQL? An Introduction to Hadoop
Dan Harvey
 
PDF
hadoop @ Ibmbigdata
Eric Baldeschwieler
 
PPTX
Galaxy of bits
Michal Zylinski
 
PPTX
10c introduction
mapr-academy
 
PPTX
10c introduction
Inyoung Cho
 
PPTX
Big Data & Hadoop Introduction
Jayant Mukherjee
 
PDF
Research on big data
Roby Chen
 
PDF
Research ON Big Data
mysqlops
 
PDF
Common and unique use cases for Apache Hadoop
Brock Noland
 
PDF
Commonanduniqueusecases 110831113310-phpapp01
eimhee
 
PDF
Omaha Java Users Group - Introduction to HBase and Hadoop
Shawn Hermans
 
PPTX
NYC-Meetup- Introduction to Hadoop Echosystem
AL500745425
 
PPTX
hadoop 101 aug 21 2012 tohug
Adam Muise
 
PDF
Keynote from ApacheCon NA 2011
Hortonworks
 
PPTX
Hadoop For Enterprises
nvvrajesh
 
PPTX
Seattle Scalability Meetup - Ted Dunning - MapR
clive boulton
 
PDF
Building apps with HBase - Data Days Texas March 2013
amansk
 
PPTX
Unlocking value in your (big) data
Oscar Renalias
 
PDF
Cloud computing era
TrendProgContest13
 
PDF
Searching conversations with hadoop
DataWorks Summit
 
Data Processing in the Work of NoSQL? An Introduction to Hadoop
Dan Harvey
 
hadoop @ Ibmbigdata
Eric Baldeschwieler
 
Galaxy of bits
Michal Zylinski
 
10c introduction
mapr-academy
 
10c introduction
Inyoung Cho
 
Big Data & Hadoop Introduction
Jayant Mukherjee
 
Research on big data
Roby Chen
 
Research ON Big Data
mysqlops
 
Common and unique use cases for Apache Hadoop
Brock Noland
 
Commonanduniqueusecases 110831113310-phpapp01
eimhee
 
Omaha Java Users Group - Introduction to HBase and Hadoop
Shawn Hermans
 
NYC-Meetup- Introduction to Hadoop Echosystem
AL500745425
 
hadoop 101 aug 21 2012 tohug
Adam Muise
 
Keynote from ApacheCon NA 2011
Hortonworks
 
Hadoop For Enterprises
nvvrajesh
 
Seattle Scalability Meetup - Ted Dunning - MapR
clive boulton
 
Building apps with HBase - Data Days Texas March 2013
amansk
 
Unlocking value in your (big) data
Oscar Renalias
 
Cloud computing era
TrendProgContest13
 
Searching conversations with hadoop
DataWorks Summit
 

Recently uploaded (20)

PPTX
From Sci-Fi to Reality: Exploring AI Evolution
Svetlana Meissner
 
PPTX
Webinar: Introduction to LF Energy EVerest
DanBrown980551
 
PDF
Achieving Consistent and Reliable AI Code Generation - Medusa AI
medusaaico
 
PPTX
AUTOMATION AND ROBOTICS IN PHARMA INDUSTRY.pptx
sameeraaabegumm
 
PDF
Advancing WebDriver BiDi support in WebKit
Igalia
 
PDF
LOOPS in C Programming Language - Technology
RishabhDwivedi43
 
PDF
Using FME to Develop Self-Service CAD Applications for a Major UK Police Force
Safe Software
 
PDF
"Beyond English: Navigating the Challenges of Building a Ukrainian-language R...
Fwdays
 
PDF
CIFDAQ Market Insights for July 7th 2025
CIFDAQ
 
PDF
Mastering Financial Management in Direct Selling
Epixel MLM Software
 
PDF
Transcript: New from BookNet Canada for 2025: BNC BiblioShare - Tech Forum 2025
BookNet Canada
 
PPTX
OpenID AuthZEN - Analyst Briefing July 2025
David Brossard
 
PDF
Exolore The Essential AI Tools in 2025.pdf
Srinivasan M
 
PPTX
The Project Compass - GDG on Campus MSIT
dscmsitkol
 
PDF
Empower Inclusion Through Accessible Java Applications
Ana-Maria Mihalceanu
 
PDF
“NPU IP Hardware Shaped Through Software and Use-case Analysis,” a Presentati...
Edge AI and Vision Alliance
 
PDF
Staying Human in a Machine- Accelerated World
Catalin Jora
 
PDF
CIFDAQ Token Spotlight for 9th July 2025
CIFDAQ
 
PDF
Building Real-Time Digital Twins with IBM Maximo & ArcGIS Indoors
Safe Software
 
PDF
Reverse Engineering of Security Products: Developing an Advanced Microsoft De...
nwbxhhcyjv
 
From Sci-Fi to Reality: Exploring AI Evolution
Svetlana Meissner
 
Webinar: Introduction to LF Energy EVerest
DanBrown980551
 
Achieving Consistent and Reliable AI Code Generation - Medusa AI
medusaaico
 
AUTOMATION AND ROBOTICS IN PHARMA INDUSTRY.pptx
sameeraaabegumm
 
Advancing WebDriver BiDi support in WebKit
Igalia
 
LOOPS in C Programming Language - Technology
RishabhDwivedi43
 
Using FME to Develop Self-Service CAD Applications for a Major UK Police Force
Safe Software
 
"Beyond English: Navigating the Challenges of Building a Ukrainian-language R...
Fwdays
 
CIFDAQ Market Insights for July 7th 2025
CIFDAQ
 
Mastering Financial Management in Direct Selling
Epixel MLM Software
 
Transcript: New from BookNet Canada for 2025: BNC BiblioShare - Tech Forum 2025
BookNet Canada
 
OpenID AuthZEN - Analyst Briefing July 2025
David Brossard
 
Exolore The Essential AI Tools in 2025.pdf
Srinivasan M
 
The Project Compass - GDG on Campus MSIT
dscmsitkol
 
Empower Inclusion Through Accessible Java Applications
Ana-Maria Mihalceanu
 
“NPU IP Hardware Shaped Through Software and Use-case Analysis,” a Presentati...
Edge AI and Vision Alliance
 
Staying Human in a Machine- Accelerated World
Catalin Jora
 
CIFDAQ Token Spotlight for 9th July 2025
CIFDAQ
 
Building Real-Time Digital Twins with IBM Maximo & ArcGIS Indoors
Safe Software
 
Reverse Engineering of Security Products: Developing an Advanced Microsoft De...
nwbxhhcyjv
 
Ad

An Introduction to Hadoop

  • 1. An Introduction to Hadoop By Dan Harvey
  • 2. “The Apache Hadoop project develops open-source software for reliable, scalable, distributed computing”
  • 3. Into the past • Doug Cutting: Lucene search library • Linear merge sorted indexes • Disk was the bottleneck • How to speed and scale up?
  • 4. Split over more disks... Then more machines...
  • 5. Jim Grey on Disks Throughput on SATA disk 49.1 MBps 50 37.5 Throughput in MBps t Random 25 Sequential 12.5 0.6 MBps 0 Access Type Barclay, T., Chong, W., & Gray, J. (2003). A Quick Look at Serial ATA Disk Performance.
  • 6. Jim Grey on Disks Time to read 2TB disk 50 40.5 days 37.5 Time in Days 0.6 MBps 25 49.1 MBps 12.5 0.5 days 0 Throughput
  • 7. So... • Use more disks & machines • Use sequential disk access • Linear merge == sequential access! • But how to make it accessible?
  • 8. MapReduce: Simplified Data Processing on Large Clusters Jeffrey Dean and Sanjay Ghemawat [email protected], [email protected] Google, Inc. Abstract given day, etc. Most such computations are conceptu- ally straightforward. However, the input data is usually MapReduce is a programming model and an associ- large and the computations have to be distributed across ated implementation for processing and generating large hundreds or thousands of machines in order to finish in data sets. Users specify a map function that processes a a reasonable amount of time. The issues of how to par- key/value pair to generate a set of intermediate key/value allelize the computation, distribute the data, and handle pairs, and a reduce function that merges all intermediate failures conspire to obscure the original simple compu- values associated with the same intermediate key. Many tation with large amounts of complex code to deal with real world tasks are expressible in this model, as shown these issues. in the paper. As a reaction to this complexity, we designed a new Programs written in this functional style are automati- abstraction that allows us to express the simple computa- cally parallelized and executed on a large cluster of com- tions we were trying to perform but hides the messy de- modity machines. The run-time system takes care of the tails of parallelization, fault-tolerance, data distribution Google’s MapReduce details of partitioning the input data, scheduling the pro- and load balancing in a library. Our abstraction is in- gram’s execution across a set of machines, handling ma- spired by the map and reduce primitives present in Lisp chine failures, and managing the required inter-machine and many other functional languages. We realized that communication. This allows programmers without any most of our computations involved applying a map op- experience with parallel and distributed systems to eas- eration to each logical “record” in our input in order to ily utilize the resources of a large distributed system. compute a set of intermediate key/value pairs, and then Our implementation of MapReduce runs on a large applying a reduce operation to all the values that shared the same key, in order to combine the derived data ap-
  • 10. Distributed Storage MapReduce: Simplified Data Processing on Large Clusters Jeffrey Dean and Sanjay Ghemawat MapReduce: Simplified Data Processing on Large Clusters [email protected], [email protected] Google, Inc. Jeffrey Dean and Sanjay Ghemawat MapReduce: Simplified Data Processing on Large Clusters [email protected], [email protected] Abstract given day, etc. Most such computations are conceptu- Google, Inc. ally straightforward. However, the input data is usually Jeffrey Dean and Sanjay Ghemawatmodel and an associ- MapReduce is a programming large and the computations have to be distributed across ated implementation for processing and generating large hundreds or thousands of machines in order to finish in [email protected], [email protected] function that processes a data sets. Users specify a map MapReduce: Simplified Data Processing on Large Clusters key/value pair to generate a set of intermediate key/value a reasonable amount of time. The issues of how to par- allelize the computation, distribute the data, and handle Abstract a reduce function that merges all intermediate such computations are conceptu- pairs, Google, Inc. and given day, etc. Most failures conspire to obscure the original simple compu- values associated with the same intermediate key. ManyHowever, the input data is usually MapReduce is a programming model and an associ- ally straightforward. tation with large amounts of complex code to deal with Jeffrey Dean and Sanjay Ghemawat real world tasks are expressible largemodel, computations have to be distributed across this and the ated implementation for processing and generating large in hundreds or as shown of these issues. order to finish in thousands machines in data sets. Users specify a in thefunction that processes a map paper. a reasonable amount of time. Thereaction to thisto par- As a issues of how complexity, we designed a new [email protected], [email protected] Abstract Programs written key/value Most such are automati- areabstraction that allows given day, etc. computations conceptu- key/value pair to generate a set of intermediate in this functional stylethe computation, distribute the data, and us to express the simple computa- allelize handle ally straightforward.large clusterthe com- data is usually However, of input Google, Inc. MapReduce is apairs, and a reduce function that merges all intermediate on a programming model and cally parallelized andand the computations conspirebe distributed across trying to perform but hides the messy de- an associ- large executed failures have to to obscure thewe were simple compu- tions original values associated with the modity machines. The run-time system takes care of the same intermediate key. Many ated implementation for processing and generating large tation with large amounts tails of parallelization, fault-tolerance, data distribution of finish in code to deal with complex hundreds or thousands of machines in order to data sets. Users specifyworld tasks are expressible in this model, as shown data, scheduling the pro- of how to par- real a map function that processes partitioning the input these issues. details of a and load balancing in a library. Our abstraction is in- gram’s execution reasonable amount of time. The issues key/value pair to generatepaper.of intermediate key/value in the a set a across a set of machines, handling ma- As distribute to data, and handle map and reduce primitives present in Lisp spired by the allelize the computation, a reaction thethis complexity, we designed a new chine failures, and managing the required inter-machine pairs, and a reduce function that merges all intermediate style are automati- toabstraction that allowssimple compu-other functional languages. We realized that and manythe simple computa- Abstract given day, etc. Most such computations are conceptu- Programs written in this functional conspire obscure the original us to express failuresallows Replicated Blocks communication. This of com- programmers without any values associated with the same intermediate key. on a large cluster with large amounts of complex code to deal but hides the messy involved applying a map op- cally parallelized and executed Many most of our computations de- ally straightforward. However, the input data is usually tation tions we were trying to perform with MapReduce is a programming model and an associ- large and the computations have to be distributed across real world tasks are expressible in this model, as shown withthese issues. distributedof parallelization, fault-tolerance, data distribution in our input in order to experience parallel and modity machines. The run-time system takes care of the tails systems to eas- eration to each logical “record” ated implementation for processing and generating large in the paper. ily utilize the resources ofpro- a large distributed system. details of partitioning the input data, scheduling a reaction to and load balancing in a library. new set of intermediate key/value pairs, and then compute a hundreds or thousands of machines in order to finish in As the this complexity, we designed a Our abstraction is in- data sets. Users specify a map function that processes a a reasonable amount of time. The issues of how to par- gram’s functional style a setOurmachines, handling ma- allows usrunsthe map and reduce primitives present in Lisp all the values that shared Programs written in thisexecution across are automati-of implementation of that abstraction spired to express the simple computa- reduce operation to MapReduce by on a large applying a key/value pair to generate a set of intermediate key/value allelize the computation, distribute the data, and handle cally parallelized and executed on and managing of com-commodity machines and to many other functional messy de- on Largethat chine failures, a large cluster the requiredtions we were trying is performscalable:Processing We realized combine the derived data ap- cluster of MapReduce: Simplified Data inter-machine and highly but hides thethe same key, in order to Clusters languages. pairs, and a reduce function that merges all intermediate failures conspire to obscure the original simple compu- communication. This allows programmers without any care of the most of our computations propriately. Our use mapaop- modity machines. The run-time system takes a typical MapReduce computation processes many ter- involved applying a of functional model with user- tails of parallelization, fault-tolerance, data distribution values associated with the same intermediate key. Many tation with large amounts of complex code to deal with experience data, scheduling distributed on thousands of machines. Programmers “record” in map and in order to abytes of data details of partitioning the inputwith parallel and the pro- systems to balancingeration to each Our abstraction is in- and load eas- in a library. logical specified our input reduce operations allows us to paral- real world tasks are expressible in this model, as shown these issues. a large system easy to by the map Jeffrey Dean and Sanjay in large computations then and to use re-execution gram’s execution across a set the machines, of find thema- spired use: MapReduce intermediate key/value pairs, and easily ily utilize of resources handling distributed system. hundreds ofreduce set of pro- present Ghemawat compute a primitives and lelize Lisp in the paper. As a reaction to this complexity, we designed a new MapReduce: Simplified Data Processing one thou- as the primary grams have been implemented and applying a on Large Clustersvalues that shared fault tolerance. upwards ofreduce operation to all the mechanism for chine failures, and managing the required inter-machine runs on a large functional languages. We realized that Our implementation of MapReduce and many other Programs written in this functional style are automati- abstraction that allows us to express the simple computa- sand MapReduce jobs are executed [email protected], [email protected] contributions ap-this work are a simple and cluster of programmers without any Google’s clusters to combine the derived data of the same key, in order communication. This allows commodity machines and is highly of our computations involved applying a map op- The major most scalable: cally parallelized and executed on a large cluster of com- tions we were trying to perform but hides the messy de- experience with parallel andMapReduce systems today.processes many each logical “record” Our use of apowerfulto model that enables automatic parallelization every eas- a typical distributed computation eration to ter- propriately. in our input in order interface with user- functional modity machines. The run-time system takes care of the tails of parallelization, fault-tolerance, data distribution ily utilize the resources of oflarge distributed system. specified mapGoogle, Inc. distribution of large-scale computations, combined and abytes a data on thousands of machines.Dean and set of intermediate key/value pairs, and then allows us to paral- Jeffrey compute a Sanjay Ghemawatand reduce with an implementation of this interface that achieves Programmers operations details of partitioning the input data, scheduling the pro- and load balancing in a library. Our abstraction is in- Our implementation the system easy to use:1hundreds of MapReduceapro- find of MapReduce runs on a large lelize large computations easily and to use re-execution applying reduce operation to all the values that shared Introduction gram’s execution across a set of machines, handling ma- spired by the map and reduce primitives present in Lisp MapReduce: Simplified scalable: the one key, Large Clusters derived Section 2 describeslargebasic programming model and cluster of commodity machines and is highlyData upwards ofsame thou- in order toprimary mechanism for fault tolerance. grams have been implemented and Processing on as the combine the [email protected], [email protected] high performance on data ap- the clusters of commodity PCs. chine failures, and managing the required inter-machine and many other functional languages. We realized that sand MapReduceprocessesexecuted on Google’s clusters useThe a functional model with user- are a simple and a typical MapReduce computation jobs are many ter- propriately. Our of major contributions of this work communication. This allows programmers without any most of our computations involved applying a map op- every day. abytes of data on thousands of machines. Programmers AbstractInc. Google, Over the past five years, mapauthors and many othersthat enables paral- Most such computations are conceptu- specified the and powerful interface allows us to automatic parallelization reduce operations at given day, etc. examples. Section 3 describes an imple- gives several Google have implemented computationsspecial-purpose ally straightforward.MapReduce interface tailored towards lelize large hundreds of easily andof use re-executionof the combined mentation experience with parallel and distributed systems to eas- eration to each logical “record” in our input in order to find the system easy to use: hundreds of MapReduce pro- is a programming model and an associ- large-scale computations,However, the input data is usually and distribution to Jeffrey Dean and Sanjay Ghemawat mechanism for fault tolerance.ourand interface that achieves to be distributed acrossde- MapReduce that process large amounts of raw data, large cluster-based computing environment. Section 4 ily utilize the resources of a large distributed system. compute a set of intermediate key/value pairs, and then computations as the primary with an implementation of this the computations have grams have been implemented and upwards of one thou- Our implementation of MapReduce runs on a large applying a reduce operation to all the values that shared 1 Introduction ated implementationdocuments, contributions of thisetc.,on largescribes and such as crawled for processing and generating large to hundreds several refinements of the programming model [email protected],Users The map web high performance request logs, sand MapReduce jobs are executed on Google’s clusters specify amajor function that processes a are a simple orof commodity machines in order to finish in work clusters thousands of PCs. data sets. [email protected] data, such as inverted a reasonable amount ofuseful. The issues of how to par- compute various kinds of cluster of commodity machines and is highly scalable: the same key, in order to combine the derived data ap- every day. Abstract given day, etc. 2 describes parallelization are conceptu- Section 5 has performance Section automatic computations found time. that we have powerful interface that enables Most suchthe basic programming model and key/value pair to generate a set of intermediate key/value a typical MapReduce computation processes many ter- propriately. Our use of a functional model with user- Over the past five years, the authors and many others at allyoflarge-scale computations,measurements of usually indices, various representations gives severalstructure allelize inputdescribes an distribute the data, and variety of the graph examples. Section Google, Inc. summaries of straightforward. However,tasks.the3computation, imple- combineddata is the our implementation for a handle pairs, and documents, distribution of and MapReduce is a programming web a reduce function that merges allthe computations have tointerface tailored towardsoriginalMapReduce within intermediate abytes of data on thousands of machines. Programmers specified map and reduce operations allows us to paral- Google have implementedof model andspecial-purpose largethe number of pages failures beSection to explores the use of simple compu- hundreds of an associ- with an implementation of this the MapReduce conspire obscure and mentation of interface that achieves distributed across the 6 lelize large computations easily and to use re-execution 1 Introduction ated computations that process large amounts of the same most our cluster-based computing environment. Section 4 complex code toitdeal with implementation for processingassociated with raw data,intermediate key. Many of machines in includingfinish in de- values andper host, the set of crawled generating large frequentthousands a tation with order amounts of hundreds or queries in Google large to our experiences in using as the basis find the system easy to use: hundreds of MapReduce pro- high performance on large clusters of commodity PCs. as the primary mechanism for fault tolerance. data such as crawled documents, web tasks arelogs, etc., toin reasonableseveral refinements of issues of how to par- sets. Users specify a map functionrequest expressible a this model,amount of time. The the programming model real world that processes a scribes as shown these issues. grams have been implemented and upwards of one thou- Section 2 describes the basic programming model and key/value pairvarious kinds setthe intermediate key/value to generate a in derived data, such as invertedallelize the computation, useful. Section 5 has this complexity, we designed a new of paper. sand MapReduce jobs are executed on Google’s clusters The major contributions of this work are a simple and Over the past five Abstractauthors and many others at given day,several examples.we have found distribute imple- to performance compute years, the of that Section 3 describes ana the As reaction gives etc. Most such computations are conceptu- data, and handle pairs, and a reduce function that merges written in this functional measurements of our implementation allows variety of the simple computa- Programs all intermediate style are automati- every day. powerful interface that enables automatic parallelization Google have implemented hundredsrepresentations of the graph structure failures conspire on Operating usually that for andus to express of MapReduce abstraction indices, various of special-purpose ally mentation ’04:the However, theinterface tailoredoriginal simple a Implementation straightforward.a large cluster ofto obscureisthe towards Design compu- values associated with and USENIX Association executed on6th Symposium inputexplores Systemsof MapReduce withinbut hides the messy de- 137 the same intermediate and OSDI cally parallelized key. Many MapReduce is athat of web documents, summaries of the number the pages computing to com- data large be distributed thewe 4 de- trying to with tation with Section 6amounts ofSection were to deal perform tions use code complex and distribution of large-scale computations, combined computations programming model an associ- process large amounts of raw data, and of computations have environment. across our cluster-based tasks. with an implementation of this interface that achieves real crawled per and generating in this largeThe run-time ated such as crawled for processing are expressible large model,scribes severalthese issues. including our totails of in in using it as fault-tolerance, data distribution world tasks host, the set of most machines. as shown implementation documents, web request modityetc., frequent queries in asystem takes carethe order experiences Google of in theof parallelization, the basis logs, to hundreds or thousands of machines refinements programming model finish 1 Introduction high performance on large clusters of commodity PCs. data compute various theapaper.derived data, such as inverteda reasonable have data, schedulingSection 5 complexity, we designed aanew in sets. Users specify map function that processesof partitioning the input found time. The the pro- hasand load balancing in library. Our abstraction is in- kinds of details a As a reaction to this that we amount of useful. issues of how to par- performance Section 2 describes the basic programming model and key/value pair to generate a set of intermediatefunctional style are measurementsmachines, handling ma- forspired by the map and reduce primitives present in Lisp indices, various representations of gram’s execution across a set of abstraction that allows us to express the simple computa- Programs written inthe graph structure this key/value automati- allelize the computation, distribute the data, anda handle of of our implementation variety Over the past five years, the authors and many others at gives several examples. Section 3 describes an imple- pairs, and a reduce cally parallelized of the number of large cluster of com- the tions we wereoriginalto perform many other functional languages. We realized that function that merges executed on failures,failures conspire to required inter-machine of web documents, summaries and all intermediate chine a pages and managing and but hides tasks. Section 6 explores thetrying simple compu- obscure the use of MapReduce within the messy de- Google have implemented hundreds of special-purpose mentation of the MapReduce interface tailored towards values associated host, the same intermediate key.’04: 6thin takes This of the programmerscomplex inand to most with data distribution modity set of most frequent queries Symposium allows tails Systems Design using it as of basis USENIX Association OSDI system communication. care on Operating of parallelization, fault-tolerance, without any Implementation crawled per with the machines. The run-timeMany a tation with large amounts experiences code deal the our computations involved applying a map op- Google including our of 137 computations that process large amounts of raw data, our cluster-based computing environment. Section 4 de- real world tasks are expressible in this model, experience with parallelpro- distributed systems to eas-a library. Our each logical is in- details of partitioning the inputshownscheduling the and as data, these issues. a largeand load balancing in eration to abstraction “record” in our input in order to in the paper. gram’s execution across a set of machines, handlingof ily utilize the resources ma- distributed system. compute a set of intermediate key/value pairs, and then such as crawled documents, web request logs, etc., to scribes several refinements of the programming model As a reaction to spired by the mapwe designed primitives present in Lisp this complexity, and reduce a new compute various kinds of derived data, such as inverted that we have found useful. Section 5 has performance chine failures, and managing the required inter-machine Programs written in this functional style are automati- and many other the simple computa-a reduce operation to all the values that shared Our implementation of MapReduce runs on a large applying abstraction that allows us to express functional languages. We realized that communication. This allows programmers without any the same key, 137in order to combine the derived data ap- indices, various representations of the graph structure measurements of our implementation for a variety of cally parallelized and executed’04:a6th Symposium com- commodity machines mostis Implementation the involved applying a map op- USENIX Association OSDI on large cluster of on of cluster Operating Systems Design andofhighly scalable: and tions we were trying to performcomputations messy de- our but hides of web documents, summaries of the number of pages tasks. Section 6 explores the use of MapReduce within experience with parallel and distributed systems computation processes many ter- “record” in our input in orderato modity machines. The run-time system takes care of the MapReduceto eas- a typical eration to each logical propriately. Our use of functional model with user- tails of parallelization, fault-tolerance, data distribution ily utilize the resources of a abytesdistributed thousands of machines. a set of intermediate key/value pairs, reduce operations allows us to paral- details of partitioning the input data, scheduling the pro- dataandsystem. large of on Programmers specified map and and then crawled per host, the set of most frequent queries in a Google including our experiences in using it as the basis load balancingcompute in a library. Our abstraction is in- find ma- applying a primitives present in Lisp computations easily and to use re-execution spired a the map and reduce reduce Our implementation of MapReduce runs on by large pro- lelize large gram’s execution across a set of machines, handlingthe system easy to use: hundreds of MapReduceoperation to all the values that shared as the primary mechanism for fault tolerance. chine failures, and cluster of commodity machines and is been implemented functional languages. Weto combine the derived data ap- managing the required inter-machine highlymany other and upwards of one order realized that grams have and scalable: the same key, in thou- communication. This allows programmers without MapReduce jobs our executed on Google’s clusters a afunctional model with user- this work are a simple and sand any a typical MapReduce computation processes many ter- most of are propriately. Our use of computations involved applying map op- The major contributions of USENIX Association OSDI ’04: 6th Symposium on Operating Systems Design and Implementation 137 every day. specified map and reduce operations interface thatparal- automatic parallelization powerful allows us to enables
  • 11. Data Locality: Map MapReduce: Simplified Data Processing on Large Clusters Jeffrey Dean and Sanjay Ghemawat MapReduce: Simplified Data Processing on Large Clusters [email protected], [email protected] Google, Inc. MapReduce, Simplified, Data, Processing, Jeffrey Dean and Sanjay Ghemawat MapReduce: Simplified Data Processing on Large Clusters [email protected], [email protected] Large, Clusters, Jeffrey, Dean, and, Sanjay, Google, Inc. Jeffrey Dean and Sanjay Ghemawat [email protected], [email protected] Ghemawat, Google, Inc, ... Abstract Google, Inc. given day, etc. Most such computations are conceptu- ally straightforward. However, the input data is usually MapReduce is a programming model and an associ- large and the computations have to be distributed across ated implementation for processing and generating large hundreds or thousands of machines in order to finish in data sets. Users specify a map function that processes a a reasonable amount of time. The issues of how to par- key/value pair to generate a set of intermediate key/value allelize the computation, distribute the data, and handle pairs, and a reduce function that merges all intermediate Map Stage failures conspire to obscure the original simple compu- values associated with the same intermediate key. Many MapReduce: Simplified Dataof complex code to deal with real world tasks are expressible in this model, as shown tation with large amounts Processing on Large Clusters these issues. in the paper. As a reaction to this complexity, we designed a new Programs written in this functional style are automati- abstraction that allows us to express the simple computa- cally parallelized and executed on a large cluster of com- Jeffrey Dean and Sanjay Ghemawat tions we were trying to perform but hides the messy de- modity machines. The run-time system takes care of the tails of parallelization, fault-tolerance, data distribution [email protected], [email protected] details of partitioning the input data, scheduling the pro- and load balancing in a library. Our abstraction is in- gram’s execution across a set of machines, handling ma- spired by the map and reduce Inc. Google, primitives present in Lisp chine failures, and managing the required inter-machine and many other functional languages. We realized that communication. This allows programmers without any most of our computations involved applying a map op- experience with parallel and distributed systems to eas- eration to each logical “record” in our input in order to ily utilize the resources of a large distributed system. compute a set of intermediate key/value etc. Most then computations are conceptu- Abstract given day, pairs, and such Our implementation of MapReduce runs on a large applying a reduce operation to all straightforward.shared ally the values that However, the input data is usually cluster of commodity machinesMapReduce is a programming model key, an order to combineand the computations have to be distributed across and is highly scalable: the same and in associ- large the derived data ap- ated implementation for processing and generating large a functional model with user-machines in order to finish in a typical MapReduce computation processes many ter- propriately. Our use of hundreds or thousands of abytes of data on thousands of machines. Programmersmap specified that etc. Most a operations allows are to paral- The issues of how to par- Abstract Users specify a data sets. function mapprocesses and reduce us MapReduce, programming, model, associ- given day, such a reasonable amount of time. computations conceptu- find the system easy to use: hundreds of MapReduce pro- set ofally straightforward. However,and to use data is usually key/value pair to generate a lelize large computations easily intermediate key/value allelize input re-execution the the computation, distribute the data, and handle grams have been is a programmingupwardsandone associ-that merges allthe mechanism forhave to be distributed across the original simple compu- MapReduce implemented and and a reduce function aslargeprimary computations failures conspire to obscure pairs, model of an thou- the and intermediate fault tolerance. sand MapReduce jobs for processingassociated with the same intermediatecontributions of tation withare a simple andin complex code to deal with values and generating large ated implementation are executed on Google’s clusters The major key. Many this work large hundreds or thousands of machines in order amounts of to finish ated, implementation, processing, every day. Users specify a map function that are expressiblepowerful interface shown time. automatic parallelization data sets. real world tasks processes a in this model, as that enables a reasonable amount of these issues. of how to par- The issues in the intermediate key/value and distribution of large-scale computations, combined key/value pair to generate a set of paper. Abstract allelize the computation, distribute the data, to this complexity, we designed a new As a reaction and handle pairs, and a reduce function that merges written day, etc. Most suchconspire to obscureabstraction that allows us to express the simple computa- Programs all intermediate with an implementation are this interface that achieves given in this functional stylecomputations of conceptu- failures are automati- 1 Introduction the cally parallelized key. Many highHowever, the on large clustersoriginal simple PCs. ally straightforward. performance input data isthe of commodity compu- usually apReduce is avalues associated with and same intermediate and executed on a large cluster ofamounts of complex code to deal with programming model an associ- tation with large com- tions we were trying to perform but hides the messy de- generating, large, data, sets, Users, ... real world tasks are expressible large model, The run-time system takes carebe distributedprogramming model and mplementation for processing and generating modity this large as shown in machines. and the computations have to of the basic across Section 2 describes the these issues. tails of parallelization, fault-tolerance, data distribution Over the past five years, the authors and many othersor thousands several examples. Section load balancing imple- hundreds at gives of machines in order toand 3 describes an in library. Our abstraction is in- finish in sets. Users specify apaper. function that processesof partitioning the input data, scheduling the pro- complexity, we designed aanew in the map details a As a reactionissues of to this Google have implemented hundreds execution across a set mentation of The MapReduce interface tailored towards gram’s of a reasonable amount of time. that allows us how to par- simple computa- Programs written in this key/value special-purpose the alue pair to generate a set of intermediatefunctional style are automati- of machines, handling ma- to spired by the map and reduce primitives present in Lisp abstraction express the allelize the computation, distributecomputing environment. Section 4 de- the data, and handle computations that process large amounts of raw data, the required inter-machine chine failures, and managing our cluster-based and many other functional languages. We realized that and a reduce function that merges all intermediate such as crawled documents, communication. This conspire scribes several original simple compu- computations involved applying a map op- failures allows to obscure the refinements web request logs, etc., to programmers without any of the of our most programming model s associated with the same intermediate key. Many tation with and amounts of complex eas- to deal with large that we systems to code compute various kinds of derived data, such as inverted distributedhave found useful. Section to has performance eration 5 each logical “record” in our input in order to world tasks are expressible in this model, experience with parallel as shown these issues. indices, various representations utilize the resources of a large distributed system. implementation afor a variety of ily of the graph structure measurements of our compute set of intermediate key/value pairs, and then paper. of web documents, summaries of the number ofapages As reaction to this complexity, we designed a of MapReduce within use new tasks. Section 6 explores the applying a reduce operation to all the values that shared ograms written in thisper host, the set are most frequent abstraction that allows us including athe experiences in using it as the basis Our implementation of MapReduce runs on large crawled functional style of automati- queries in a Google to expressour simple computa- key, in order to combine the derived data ap- cluster of commodity machines and is highly scalable: the same a typical MapReduce computation processes many ter- propriately. Our use of a functional model with user- abytes of data on thousands of machines. Programmers specified map and reduce operations allows us to paral- lelize large computations easily and to use re-execution USENIX Association OSDI ’04:find the system easy to use: hundreds of MapReduce pro- 6th Symposium on Operating Systems Design and Implementation 137 grams have been implemented and upwards of one thou- as the primary mechanism for fault tolerance. sand MapReduce jobs are executed on Google’s clusters The major contributions of this work are a simple and every day. powerful interface that enables automatic parallelization and distribution of large-scale computations, combined with an implementation of this interface that achieves 1 Introduction high performance on large clusters of commodity PCs. Section 2 describes the basic programming model and Over the past five years, the authors and many others at gives several examples. Section 3 describes an imple-
  • 12. Shuffle and, associated, clusters, data, data, dean, MapReduce, Simplified, Data, Processing, implementation, inc, jeffrey, large, large, Large, Clusters, Jeffrey, Dean, and, Sanjay, mapreduce, mapreduce, model, generating, Ghemawat, Google, Inc, ... ghemawat, google, google, ... MapReduce, programming, model, programming, processing, processing, associated, implementation, processing, sanjay, simplified, sets, users, ... generating, large, data, sets, Users, ...
  • 13. Reduce and, 1 associated, 1 clusters, 1 and, associated, clusters, data, data, dean, data, 2 implementation, inc, jeffrey, large, large, ... mapreduce, mapreduce, model, generating, mapreduce, 2 ghemawat, google, google, ... model, 1, generating, 1 ghemawat, 1 google, 2 programming, 1 processing, 2 programming, processing, processing, sanjay, 1 sanjay, simplified, sets, users, ... ...
  • 14. So Hadoop? • Map Reduce for processing • Distributed File System (HDFS) for storage
  • 16. Uses • Log analysis • Data Warehouse • Natural Language Processing • Search indexes • Machine Learning • ...
  • 17. 1.0 and beyond • Next generation Hadoop - YARN • Not just MapReduce • General resource allocation framework
  • 18. Users

Editor's Notes

  • #2: An introduction to Hadoop\nSlides will be a mix of technical content and non technical\nHigher level\nNot sure of the audiance.\n
  • #3: \n
  • #4: \n
  • #5: To achieve both speed and scale need both of these.\n
  • #6: \n
  • #7: \n
  • #8: \n
  • #9: \n
  • #10: \n
  • #11: - Data split into blocks\n - Replicated > three times on different machines\n - Fault tolerant storage\n
  • #12: - Data split into blocks\n - Replicated > three times on different machines\n - Fault tolerant storage\n
  • #13: - Data split into blocks\n - Replicated > three times on different machines\n - Fault tolerant storage\n
  • #14: - Data split into blocks\n - Replicated > three times on different machines\n - Fault tolerant storage\n
  • #15: \n
  • #16: \n
  • #17: \n
  • #18: \n
  • #19: \n
  • #20: \n