Map Reduce
Download as PPT, PDF•11 likes•3,831 views
The document provides an overview of MapReduce, including: 1) MapReduce is a programming model and implementation that allows for large-scale data processing across clusters of computers. It handles parallelization, distribution, and reliability. 2) The programming model involves mapping input data to intermediate key-value pairs and then reducing by key to output results. 3) Example uses of MapReduce include word counting and distributed searching of text.
Map Reduce
- 1. Lecture 2 – MapReduce CPE 458 – Parallel Programming, Spring 2009 Except as otherwise noted, the content of this presentation is licensed under the Creative Commons Attribution 2.5 License.http://creativecommons.org/licenses/by/2.5
- 2. Outline MapReduce: Programming Model MapReduce Examples A Brief History MapReduce Execution Overview Hadoop MapReduce Resources
- 3. MapReduce “ A simple and powerful interface that enables automatic parallelization and distribution of large-scale computations, combined with an implementation of this interface that achieves high performance on large clusters of commodity PCs.” Dean and Ghermawat, “MapReduce: Simplified Data Processing on Large Clusters”, Google Inc.
- 4. MapReduce More simply, MapReduce is: A parallel programming model and associated implementation.
- 5. Programming Model Description The mental model the programmer has about the detailed execution of their application. Purpose Improve programmer productivity Evaluation Expressibility Simplicity Performance
- 6. Programming Models von Neumann model Execute a stream of instructions (machine code) Instructions can specify Arithmetic operations Data addresses Next instruction to execute Complexity Track billions of data locations and millions of instructions Manage with: Modular design High-level programming languages (isomorphic)
- 7. Programming Models Parallel Programming Models Message passing Independent tasks encapsulating local data Tasks interact by exchanging messages Shared memory Tasks share a common address space Tasks interact by reading and writing this space asynchronously Data parallelization Tasks execute a sequence of independent operations Data usually evenly partitioned across tasks Also referred to as “Embarrassingly parallel”
- 8. MapReduce: Programming Model Process data using special map () and reduce () functions The map() function is called on every item in the input and emits a series of intermediate key/value pairs All values associated with a given key are grouped together The reduce() function is called on every unique key, and its value list, and emits a value that is added to the output
- 9. MapReduce: Programming Model How now Brown cow How does It work now brown 1 cow 1 does 1 How 2 it 1 now 2 work 1 M M M M R R <How,1> <now,1> <brown,1> <cow,1> <How,1> <does,1> <it,1> <work,1> <now,1> <How,1 1> <now,1 1> <brown,1> <cow,1> <does,1> <it,1> <work,1> Input Output Map Reduce MapReduce Framework
- 10. MapReduce: Programming Model More formally, Map(k1,v1) --> list(k2,v2) Reduce(k2, list(v2)) --> list(v2)
- 11. MapReduce Runtime System Partitions input data Schedules execution across a set of machines Handles machine failure Manages interprocess communication
- 12. MapReduce Benefits Greatly reduces parallel programming complexity Reduces synchronization complexity Automatically partitions data Provides failure transparency Handles load balancing Practical Approximately 1000 Google MapReduce jobs run everyday.
- 13. MapReduce Examples Word frequency Map doc Reduce <word,3> <word,1> <word,1> <word,1> Runtime System <word,1,1,1>
- 14. MapReduce Examples Distributed grep Map function emits <word, line_number> if word matches search criteria Reduce function is the identity function URL access frequency Map function processes web logs, emits <url, 1> Reduce function sums values and emits <url, total>
- 15. A Brief History Functional programming (e.g., Lisp) map() function Applies a function to each value of a sequence reduce() function Combines all elements of a sequence using a binary operator
- 16. MapReduce Execution Overview The user program, via the MapReduce library, shards the input data User Program Input Data Shard 0 Shard 1 Shard 2 Shard 3 Shard 4 Shard 5 Shard 6 * Shards are typically 16-64mb in size
- 17. MapReduce Execution Overview The user program creates process copies distributed on a machine cluster. One copy will be the “Master” and the others will be worker threads. User Program Master Workers Workers Workers Workers Workers
- 18. MapReduce Resources The master distributes M map and R reduce tasks to idle workers. M == number of shards R == the intermediate key space is divided into R parts Master Idle Worker Message(Do_map_task)
- 19. MapReduce Resources Each map-task worker reads assigned input shard and outputs intermediate key/value pairs. Output buffered in RAM. Map worker Shard 0 Key/value pairs
- 20. MapReduce Execution Overview Each worker flushes intermediate values, partitioned into R regions, to disk and notifies the Master process. Master Map worker Disk locations Local Storage
- 21. MapReduce Execution Overview Master process gives disk locations to an available reduce-task worker who reads all associated intermediate data. Master Reduce worker Disk locations remote Storage
- 22. MapReduce Execution Overview Each reduce-task worker sorts its intermediate data. Calls the reduce function, passing in unique keys and associated key values. Reduce function output appended to reduce-task’s partition output file. Reduce worker Sorts data Partition Output file
- 23. MapReduce Execution Overview Master process wakes up user process when all tasks have completed. Output contained in R output files. wakeup User Program Master Output files
- 24. MapReduce Execution Overview Fault Tolerance Master process periodically pings workers Map-task failure Re-execute All output was stored locally Reduce-task failure Only re-execute partially completed tasks All output stored in the global file system
- 25. Hadoop Open source MapReduce implementation http: //hadoop .apache.org/core/index.html Uses Hadoop Distributed Filesytem (HDFS) http: //hadoop .apache. org/core/docs/current/hdfs_design .html Java ssh
- 26. References Introduction to Parallel Programming and MapReduce, Google Code University http://code.google.com/edu/parallel/mapreduce-tutorial.html Distributed Systems http://code. google . com/edu/parallel/index .html MapReduce: Simplified Data Processing on Large Clusters http://labs. google . com/papers/mapreduce .html Hadoop http: //hadoop .apache.org/core/