Setting High Availability in Hadoop Cluster
0 likes3,214 views
This document discusses achieving high availability in Hadoop clusters. It begins by introducing Hadoop and its core components like HDFS, YARN, and MapReduce. It then explains the single point of failure issue with the NameNode in Hadoop 1.x. Hadoop 2.0 introduced solutions like having an active and standby NameNode that log all filesystem edits to shared storage. ZooKeeper is used for failover detection and coordination. The document also discusses securing HDFS through access control lists and using Hadoop as a data warehouse with tools like Hive, Impala, and BI tools. Hands-on sections walk through setting up high availability for HDFS and YARN.
Setting High Availability in Hadoop Cluster
- 1. www.edureka.co/hadoop-admin Setting High Availability in Hadoop Cluster
- 2. www.edureka.co/hadoop-admin What will you learn today? Hadoop: A synonym for Big Data Hadoop High Availability Hands-On: Achieving NameNode and YARN high availability Hands-On: Securing HDFS through ACL Hadoop as a Data Warehouse
- 3. www.edureka.co/hadoop-admin What is Hadoop? Apache Hadoop is an open source, scalable and reliable solution that stores and allows distributed processing of large data sets across clusters of computers using simple programming model
- 4. www.edureka.co/hadoop-admin A closer look at Apache Hadoop Apache Hadoop includes following modules : Hadoop Distributed File System (HDFS): A distributed file system Hadoop Common: The common utilities that support the other Hadoop modules Hadoop YARN: A framework for job scheduling and cluster resource management Hadoop MapReduce: A YARN-based system for parallel processing of large data sets
- 6. www.edureka.co/hadoop-admin Maintaining High Availability In Distributed Computing, failure is a norm, which means YARN should have acceptable amount of availability NameNode - No Horizontal Scale NameNode - No High Availability Data Node Data Node Data Node …. Client get Block Locations Read Data NameNode NS Block Management
- 7. www.edureka.co/hadoop-admin NameNode: Single Point of Failure Secondary NameNode NameNode Secondary NameNode: "Not a hot standby" for the NameNode Connects to NameNode every hour* Housekeeping, backup of NemeNode metadata Saved metadata can build a failed NameNode metadata metadata Single Point Failure You give me metadata every hour, I will make it secure
- 8. www.edureka.co/hadoop-admin Hadoop 2.0 Cluster Architecture: High Availability Node Manager HDFS YARN Resource Manager Shared edit logs All name space edits logged to shared NFS storage; single writer (fencing) Read edit logs and applies to its own namespace Secondary Name Node DataNode Standby NameNode Active NameNode Container App Master Node Manager DataNode Container App Master Data Node Client DataNode Container App Master Node Manager DataNode Container App Master Node Manager NameNode High Availability Next Generation MapReduce http://hadoop.apache.org/docs/stable2/hadoop-yarn/hadoop-yarn-site/HDFSHighAvailabilityWithNFS.html HDFS HIGH AVAILABILITY
- 9. www.edureka.co/hadoop-admin NN Active NN Standby DN 1 DN 2 DN n Shared storage Failover Controller Active ZK ZK ZK Failover Controller Standby Heartbeat Heartbeat Monitor s NN’s Health Monitor s NN’s Health Block Reports to Active and standby NN: Update cmds from one Sharead NN state with single writer (fencing) HDFS Cmds
- 10. www.edureka.co/hadoop-admin ZooKeeper RMState ZooKeeper RMState ZKFC Resource Manager Active ZKFC Resource Manager Passive 1. Active Node stores all state in ZKStore 2. Failure 4. Failover 3. Standby Node become active 3. ZKFC Detects failure
- 11. www.edureka.co/hadoop-admin Monitor liveness & heath zookeeper Journal Node zookeeper zookeeper Journal Node Journal Node ZookeeperFC NameNode Standby NameNode Active DataNode DataNode DataNode ZookeeperFC Zookeeper Service Shared Edits Monitor and maintain active lock Monitor and try to take active lock Monitor liveness & heath ReadWrite
- 12. www.edureka.co/hadoop-admin Hands-On Achieving HDFS and YARN High Availability
- 14. www.edureka.co/hadoop-admin What to do with Big Data?
- 15. www.edureka.co/hadoop-admin Hadoop: The Perfect Data Warehouse Free Text Images/ Videos HCatalog HiveSQL Others …ImpalaSQL Tableau CognosQlikView LogsTransaction Sensors Pentaho HDFS Files Metadata Query Engines BI Tools
- 16. www.edureka.co/hadoop-admin What a Data Warehouse is good at? Among others, a data warehouse is the foundation for a successful business intelligence program The Data Warehouse Institute www.tdwi.org
- 17. www.edureka.co/hadoop-admin Thank You … Questions/Queries/Feedback Recording and presentation will be made available to you within 24 hours