Developing a Map Reduce Application
2 likes8,758 views
The document outlines the process for developing a MapReduce application including: 1) Writing map and reduce functions with unit tests, then a driver program to run on test data. 2) Running the program on a cluster with the full dataset and fixing issues. 3) Tuning the program for performance after it is working correctly.
Developing a Map Reduce Application
- 1. Developing a MapReduce Application Dr. C.V. Suresh Babu (CentreforKnowledgeTransfer) institute
- 2. Phases of Developing a MapReduce Application • Configuration API • Configuring the Development Environment • GenericOptionsParser, Tool and ToolRunner • Writing Unit Tests • Running locally and in a cluster on Test Data • The MapReduce Web UI • Hadoop Logs • Tuning a Job to improve performance (CentreforKnowledgeTransfer) institute
- 3. Stages 1:Developing a MapReduce Application • Writing a program in MapReduce follows a certain pattern. • You start by writing your map and reduce functions, ideally with unit tests to make sure they do what you expect. • Then you write a driver program to run a job, which can run from your IDE using a small subset of the data to check that it is working. • If it fails, you can use your IDE’s debugger to find the source of the problem. • With this information, you can expand your unit tests to cover this case and improve your mapper or reducer as appropriate to handle such input correctly. (CentreforKnowledgeTransfer) institute
- 4. Stages 2:Developing a MapReduce Application • When the program runs as expected against the small dataset, you are ready to unleash it on a cluster. • Running against the full dataset is likely to expose some more issues, which you can fix as before, by expanding your tests and mapper or reducer to handle the new cases. • Debugging failing programs in the cluster is a challenge, so we look at some common techniques to make it easier. (CentreforKnowledgeTransfer) institute
- 5. Stage 3: Developing a MapReduce Application • After the program is working, you may wish to do some tuning, first by running through some standard checks for making MapReduce programs faster and then by doing task profiling. • Profiling distributed programs is not easy, but Hadoop has hooks to aid the process. (CentreforKnowledgeTransfer) institute
- 6. Example: Word Count Task: Counting the word occurances (frequencies) in a text file (or set of files). <word, count >as <key, value >pair Mapper: Emits <word, 1 >for each word (no counting at this part). Shuffle in between: pairs with same keys grouped together and passed to a single machine. Reducer: Sums up the values (1s) with the same key value (CentreforKnowledgeTransfer) institute