HDFS-HC2: Analysis of Data Placement Strategy based on Computing Power of Nodes on Heterogeneous Hadoop Clusters
Download as PPTX, PDF•0 likes•1,065 views
The document presents an analysis of data placement strategies based on the computing power of nodes in heterogeneous Hadoop clusters. It covers various aspects such as the MapReduce programming model, HDFS architecture, and a design for evaluating Hadoop distribution through applications like wordcount and grep. Results from implementing the crbalancer strategy and its effects on data placement are discussed.
HDFS-HC2: Analysis of Data Placement Strategy based on Computing Power of Nodes on Heterogeneous Hadoop Clusters
- 1. Analysis of Data Placement Strategy based on Computing Power of Nodes on Heterogeneous Hadoop Clusters Sanket Reddy Chintapalli Advisor - Dr. Xiao Qin
- 2. Presentation Overview ● Synopsis ● Mapreduce Programming Model Overview ● HDFS Overview ● Motivation ● Design ● Software Description ● Hardware Description ● Results ● Conclusion
- 3. Synopsis ● Data placement strategy ● Heterogeneous Clusters ● Computing Power ● Calculating Computing Ratio ● WordCount and Grep
- 4. MapReduce Model ● Hadoop 1.0 and Hadoop 2.0 ● Master - Slave Model ● JobTracker and TaskTracker Hadoop 1.0 ● YARN Hadoop 2.0 ● Resource Manager YARN ● Application Manager YARN ● Node Manager YARN ● MapReduce Flow
- 6. Mapreduce Model - 1.0
- 7. Mapreduce Model - YARN - 2.0
- 8. Mapreduce Model - Flow
- 9. HDFS ● Namenode ● Datanode ● Replication ● Federated Namenodes
- 12. HDFS Federated Namenodes ● Scalability ● Performance ● Isolation - overload
- 13. Motivation
- 14. Software Description ● Hadoop 2.3.0 ● Maven ● Eclipse ● Protocol Buffers
- 16. Design Run WordCount and Grep Applications on individual nodes
- 17. Design Calculate Computing Power of Individual Nodes for a specific application
- 18. Design ● Evaluate Hadoop Distribution by running grep and wordcount together on all nodes ● Run the CRBalancer to balance the nodes ● Finally re-run the applications to note the ramifications of the data placement strategy.
- 19. Design - Algorithm CRBalancer Strategy
- 20. Implementation ● CRBalancer ● CRBalancingPolicy ● CRNamenodeConnector
- 22. Results - Grep
- 23. Questions ??