Toward Better Multi-Tenancy Support from HDFS
Download as PPT, PDF•5 likes•2,156 views
This document discusses ongoing work to improve HDFS multi-tenancy support and resource management. It describes how HDFS currently supports resource sharing, isolation, and management. Improvements include fair call queueing for NameNode RPCs, throttling techniques to avoid queue overload, and a proposed resource coupon system for reserving NameNode and DataNode resources. The goal is to provide better quality of service and allow prioritization of important jobs over batch workloads.
Toward Better Multi-Tenancy Support from HDFS
- 1. 1 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Toward Better Multi- Tenancy Support from HDFS Xiaoyu Yao Email: [email protected]
- 2. 2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved About myself ⬢ Member of Technical Staff at Hortonworks since 2014 ⬢ Apache Hadoop Committer and PMC member. ⬢ Currently working on HDFS. ⬢ This talk is to help better understanding of HDFS multi-tenancy support and ongoing work for better resource management.
- 3. 3 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Agenda ⬢ Overview ⬢ Hadoop multi-tenancy features ⬢ HDFS resources and multi-tenancy offerings ⬢ HDFS resource management via resource coupon ⬢ Q&A
- 4. 4 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Overview ⬢ Centrally managed infrastructure –Consolidate to simplify management and lower TCO –Better utilization and efficiency ⬢ Requirement –Resource Sharing –Resource Isolation –Resource Control
- 5. 5 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Multi-Tenancy Support from Hadoop Resource Sharing Resource Isolation Resource Management HBASE Y Namespace, Region Server Group Quota YARN Y Queue, Node Label ... Capacity Scheduler, ... HDFS Y Federation Quota, FairCallQueue, Backoff
- 6. 6 © Hortonworks Inc. 2011 – 2016. All Rights Reserved HDFS Resources ⬢ Capacity –Namespace –Storage Space –Storage Type ⬢ Operational Resources –Namenode •RPC –Datanode •Disk & Network
- 7. 7 © Hortonworks Inc. 2011 – 2016. All Rights Reserved HDFS Resource Sharing/Isolation – Federation
- 8. 8 © Hortonworks Inc. 2011 – 2016. All Rights Reserved HDFS Capacity Management – Quota ⬢ Quota –Namespace –StorageSpace –HDFS-7584 Quota by Storage Types ⬢ Limitations –Static –Per directory –No per user/job control
- 9. 9 © Hortonworks Inc. 2011 – 2016. All Rights Reserved HDFS Operational Resource Management – Namenode RPC Isolation (1) ⬢Internal RPC –DN->NN block report, heartbeat, etc. –ZKFC->NN liveness check ⬢External RPC –Client RPCs from HDFSClients such as MR jobs/Hive queries/HBase Client Listener Reader Reader Call Queue Handler Handler Handler FSN
- 10. 10 © Hortonworks Inc. 2011 – 2016. All Rights Reserved HDFS Operational Resource Management – Namenode RPC Isolation (2) ⬢Use case: –HFDS access from normal jobs impacted by offending jobs –Internal RPCs impacted by External RPCs –One blocked RPC method could affect others ⬢Protect HDFS internal RPCs: –Dedicated service RPC server/port •Isolate DN->NN block report, heartbeat, etc. –Dedicated lifeline RPC server/port •Protect ZKFC->NN liveness check ⬢All external RPCs go to the default port (e.g., 8020)
- 11. 11 © Hortonworks Inc. 2011 – 2016. All Rights Reserved HDFS Resource Management – Name Node RPC Call Queue ⬢ In multi-tenancy scenario, call queue should play an important role like a shock absorber to accommodate different workload, converting busty arrivals into smooth, steady departures. ⬢ Good call queue –queue without call bloat –catches and handles bursts with no more than a temporary increase of queue delay –maximum server utilization ⬢ Bad call queue –queue that exhibits call bloat –queue filled up and stay filled upon bursts –low utilization and high queue latency
- 12. 12 © Hortonworks Inc. 2011 – 2016. All Rights Reserved HDFS Resource Management - Fair Call Queue ⬢ Before HADOOP-9640 LinkedBlockingQueue –Single queue –Client blocked and timeout/fail when queue is full ⬢ HADOOP-9640 - Fair Call Queue –Multiple priority levels and call queues with different processing priority –Each RPC is assigned a priority by scheduler –High priority RPC calls are put into call queue with higher probability of being executed. Scheduler Queue 0 Queue ... Queue 2 Multiplexer (WRR)
- 13. 13 © Hortonworks Inc. 2011 – 2016. All Rights Reserved HDFS Resource Management – Namenode RPC Throttling <1> ⬢ HADOOP-10597 Backoff when the call queue is full –Send back a Retriable exception –Let the client do exponential wait and retry instead of blocking/timeout/failed the call.
- 14. 14 © Hortonworks Inc. 2011 – 2016. All Rights Reserved HDFS Resource Management – Namenode RPC Throttling <2> ⬢ HADOOP-12916 Backoff based on response time –The basic idea: Backoff earlier to avoid call queue overload so that namenode can recover quickly. –Low priority calls get backed off if response time of high priority call is over predefined threshold. –More per user/queue metrics added for trouble shooting.
- 15. 15 © Hortonworks Inc. 2011 – 2016. All Rights Reserved HDFS Resource Management – Namenode RPC Throttling <3> ⬢ Abstract scheduler interface from call queue for pluggable RPC priority assignment –DefaultRpcScheduler: all RPC calls with same priority –DecayRpcScheduler: from original FairCallQueue priority assigned based on previous call volumes of users. –Other experimental schedulers: configurable list of high priority user/group for low latency jobs, medium priority user/group for normal jobs and low priority user/group for batch jobs.
- 16. 16 © Hortonworks Inc. 2011 – 2016. All Rights Reserved HDFS resource management - QoS ⬢ Use case: –Allow high performance QoS mechanism with minimum decoding effort on server side ⬢ HADOOP-9194 QoS support for Hadoop RPC –One bytes in RPC header to facilitate QoS mechanism –E.g., differentiate OLTP/OLAP, batch/streaming against the same HDFS ⬢ Limitation –No mechanism level implementation yet
- 17. 17 © Hortonworks Inc. 2011 – 2016. All Rights Reserved HDFS resource management with YARN ⬢ Use Case –Priority inversion without centralized resource management (e.g., RPC calls from high priority YARN jobs may be put into low priority HDFS namenode call queue) –Identify and manage ”bad” caller effectively ⬢ Namenode – RPC handler –FairCallQueue offers the fairness use of namenode RPC handlers –No guarantee of differentiation ⬢ Datanode – I/O bandwidth –No differentiation of writer/reader and bandwidth usage. –Datanode allows static throttling balancer I/O.
- 18. 18 © Hortonworks Inc. 2011 – 2016. All Rights Reserved HDFS Namenode Resource Reservation ⬢ HADOOP-13128 propose HDFS namenode resource reservation via resource coupon –From throttling to manage –Similar to delegation token in many aspects –Works for both Kerberos and non-Kerberos cluster –Allows only privileged service user to request resource coupons from namenode. –Coupon can be serialized/de-serialized for use within container. –Coupon can be renewed for long running jobs or canceled after the intended job is finished.
- 19. 19 © Hortonworks Inc. 2011 – 2016. All Rights Reserved HDFS Namenode Resource Coupon ⬢ Coupon Identifier –Finer grain owner (MR job ID, Hive Query ID) to help identify and manage “good” and “bad” callers –Resource type (Namenode RPC or Datanode I/O bandwidth) –Flexible management unit for different resources. •Min/Max percentage (e.g. Namenode RPC) •Absolute value (Datanode I/O bandwidth)
- 20. 20 © Hortonworks Inc. 2011 – 2016. All Rights Reserved HDFS Namenode Resource Coupon Manager (RCM) ⬢ Grant/Renew/Cancel resource coupon ⬢ Monitor and report resource usage ⬢ Check and validate resource use requests
- 21. 21 © Hortonworks Inc. 2011 – 2016. All Rights Reserved HDFS Namenode Resource Pool HDFS Namenode Resource Pool Fairness Pool Managed Pool Applications supporting Resource Coupon (YARN/HBASE) Legacy Applications without Resource Coupon
- 22. 22 © Hortonworks Inc. 2011 – 2016. All Rights Reserved HDFS Namenode Resource Coupon Manager (RCM) NEW Client YARN Resource Manager HDFS Namenode RCM HDFS Datanode YARN Node Manager YARN Container
- 23. 23 © Hortonworks Inc. 2011 – 2016. All Rights Reserved HDFS Resource Management – Datanode ⬢ Use case: –When a client writes to HDFS faster than the disk bandwidth of the DNs, it saturates the disk bandwidth and put the DNs into an unresponsive state. –The client only backs off by aborting / recovering the pipeline, which causes failed writes and unnecessary pipeline recovery. ⬢ Static I/O Throttling –HDFS-7265 Support HDFS IO throttling –HDFS-9796 Use a throttler for replica write in datanode –HDFS-4412 Add throttler for datanode bandwidth –HADOOP-10410 datanode Qos via ioprio_set on DataXceiver thread ⬢ Dynamic I/O Throttling –HDFS-7270 Add congestion signaling capability to DataNode write pipline(ECN) ⬢ Future work: I/O bandwidth reservation with resource coupon
- 24. 24 © Hortonworks Inc. 2011 – 2016. All Rights Reserved24 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Thank you! Q&A