Apache Hadoop YARN: state of the union

1 © Hortonworks Inc. 2011–2018. All rights reserved
Apache Hadoop YARN:
State of the Union
Sanjay Radia
Founder, Chief Architect Hortonworks
Apache Hadoop PMC

About the Speaker
• Sanjay Radia
• Chief Architect, Founder, Hortonworks
• Apache Hadoop PMC and Committer
• Part of the original Hadoop team at Yahoo! since 2007
• Chief Architect of Hadoop Core at Yahoo!
• Prior
• Data center automation, virtualization, Java, HA, OSs, File Systems
• Startup, Sun Microsystems, INRIA…
• Ph.D., University of Waterloo
Page 2Architecting the Future of Big Data

• Introduction
• Past
• State of Union for YARN
Agenda

A Brief Timeline from Past Year: GA Releases
2.8.0 2.9.0 3.0.0 3.1.0
• GPU/FPGA
• YARN Native
Service
• Placement
Constraints
• YARN Federation
• Opportunistic
Container
(Backported from 3.0)
• New YARN UI
• Timeline V2
• Global Scheduling
• Multiple Resource types
• New YARN UI
• Timeline service V2
Ever involving requirements (computation intensive, larger, services)
• Application Priority
• Reservations
• Node labels
improvements
22 March ’17 17 Nov ’17 13 Dec ‘17 02 Aug ‘182.8.4 2.9.1 3.0.3 3.1.1

Apache Hadoop 3.0/3.1

Key Themes
Scale
Platform Themes Workload Themes
Scheduling Usability Containers Resources Services

Key Themes
Scale

• Many customers with clusters made up of large number of nodes
• Oath (Yahoo!), Twitter, LinkedIn, Microsoft, Alibaba etc.
• Previous largest clusters: 6K-8K
• Now: 50K nodes in a single cluster of Microsoft [1]
• Roadmap: To 100K and beyond
[1] https://azure.microsoft.com/en-us/blog/how-microsoft-drives-exabyte-analytics-on-the-world-s-largest-yarn-cluster/
Looking at the Scale!

• Enables applications to scale to 100k of thousands of nodes
• Federation divides a large (10-100k nodes) cluster into smaller units called sub-clusters
• Federation negotiates with sub-clusters RM’s and provide resources to the application
• Applications can schedule tasks on any node
YARN Federation

Key Themes
Scale

• YARN-5139
• Problems
• Current design of one-node-at-a-time allocation cycle can lead to suboptimal decisions.
• Performance and also hard to do global placement policies
• Several coarse grained locks
• With this, we improved to
• Look at several nodes at a time
• Fine grained locks
• Multiple allocator threads
• YARN scheduler can allocate 3k+ containers per second ≈ 10 mil allocations / hour!
• 10X throughput gains
• Much better placement decisions
Moving Towards Global & Fast Scheduling

Better Placement Strategies (YARN-6592)
• Past
• Supported constraints in form of Node Locality
• Now YARN can support a lot more use cases
• Co-locate the allocations of a job on the same rack (affinity)
• Spread allocations across machines (anti-affinity) to minimize resource interference
• Allow up to a specific number of allocations in a node group (cardinality)

Better Placement Strategies (YARN-6592)
Affinity Anti-affinity
Region
Server
Region
Server

Absolute Resources Configuration in CS – YARN-5881
• The cloud model! “Give me X resources, not X%”
• Gives ability to configure Queue resources as below
<memory=24GB, vcores=20, yarn.io/gpu=2>
• Enables admins to assign different quotas of different resource-types
• No more “Single percentage value limitation for all resource-types”
root
(memory=60G, vcores=40,
gpu=10)
sales
gpu=6)
marketing
gpu=4)
engineering
gpu=0)

Useful Scheduling Features from Recent 2.X Releases
• Application Priorities
• Queue Priorities
• Previously : Resources to least satisfied queue
• Queues can have priorities e.g. (production over dev)

Key Themes
Scale

Usability: UI (1)

Usability: UI (2)

Timeline Service 2.0
• Application History
• Where did my container run
• Why is my app slow
• Why is failing
• What happened with my app
• Cluster History
• Which user had most utilization
• Largest App
• What happened in my cluster
• …
• Scalable & Robust Service
• Using HBase as backend for better scale
• More robust storage fault tolerance
• Migration and compatibility with v.1.5

Usability: Queue & Logs
API based queue management
Decentralized
(YARN-5734)
Improved logs
management
(YARN-4904)
Live application logs

Key Themes
Scale

• Better Packaging model
• Lightweight mechanism for packaging and resource isolation
• Popularized and made accessible by Docker
• Native integration ++ in YARN
• Support for “Container Runtimes” in LCE: YARN-3611
• Native process runtime
• Docker runtime
• Many security/usability improves added to 3.0.x / 3.1.x
Containers

• Run both with and without docker on the same cluster
• Choose at run-time!
Containers

• Apache Spark applications have a complex set of required software dependencies
• Docker on YARN helps to solve Spark package isolation issues with
• PySpark - Python versions, packages
• R - packages
Spark on Docker in YARN

Hadoop Apps
YARN on YARN… Towards a Private Cloud …YCloud
YARN
MR Tez Spark
Tensor
Flow
YARN
MR Tez Spark
Spark

YCloud: YARN Based Container Cloud We Use for Testing
ü Testing Hadoop on Hadoop!

Key Themes
Scale

• YARN supported only Memory
and CPU
• Now
• A generalized vector for all resources
• Admin could add arbitrary resource types!
Resource Profiles and Custom Resource Types
• Ease of resource requesting model
using profiles for apps
Profile Memory CPU GPU
Small 2 GB 4 Cores 0 Cores
Medium 4 GB 8 Cores 0 Cores
Large 16 GB 16 Cores 4 CoresMemory
CPU
GPU
FPGA
Node Manager

• Why?
• No need to setup separate clusters
• Leverage shared compute!
• Why need isolation?
• Multiple processes use the single GPU will be:
• Serialized.
• Cause OOM easily.
• GPU isolation on YARN:
• Granularity is for per-GPU device.
• Use cgroups / docker to enforce isolation.
GPU Support on YARN
Tensorﬂow 1.2
Nginx AppUbuntu 14:04
Nginx AppHost OS
GPU Base Lib v1
Volume Mount
CUDA Library 5.0

• FPGA isolation on YARN: .
• Granularity is for per-FPGA device.
• Use Cgroups to enforce the isolation.
• Currently, only Intel OpenCL SDK for FPGA is supported.
• But implementation is extensible to other FPGA SDK.
FPGA on YARN

Key Themes
Scale

Services Support in YARN
• A native YARN services framework (YARN-4692)
• Native YARN framework layer for services and beyond
• Apache Slider retired from Incubator – lessons and key code carried over to YARN
• Simplified discovery of services via DNS mechanisms: YARN-4757
• regionserver-0.hbase-app-3.hadoop.yarn.site
• Application & Services upgrades
• “Do an upgrade of my HBase app with minimal impact to end-users”
• YARN-4726

• Applications need simple APIs
• Need to be deployable “easily”
• Simple REST API layer fronting YARN
• YARN-4793 Simplified API layer
• Spawn services & Manage them
Simplified APIs for Service Definitions

LLAP on YARN
• Apache Hive LLAP is a key long running application
• Used for query processing
• Designed to run on a shared multi-tenant YARN cluster

Application Timeout – YARN-3813
• Controlling running time of workloads in YARN
• Define lifetime for an application anytime for YARN to manage.
• “Give me resources for this app/service but kill it after 15 days”
Reservation-based Scheduling: If You’re Late Don’t Blame Us! - Carlo, et al. 2015

Apache Hadoop 3.2 and Beyond

• Every user says “Give me 16GB for my task”, even though it’s only needed at peak
• Each node has some allocated but unutilized capacity. Use such capacity to run opportunistic tasks
• Preempt such tasks when needed
Container Overcommit (YARN-1011)
Un-allocated
Allocated
Utilized
Allocated-
but-unutilized
Node1
Guaranteed
Containers
Node1
Opportunistic
Containers
Preemption
When necessary

• “Take me to a node with JDK 10”
• Node Partition vs. Node Attribute
• Partition:
• One partition for one node
• ACL
• Shares between queues
• Preemption enforced.
• Attribute:
• For container placement
• No ACL/Shares on attributes
• First-come-first-serve
Node Attributes (YARN-3409) Partition - 1 (15 nodes)
Queue-A (40%)
Queue-B (60%)
Partition - 2 (15 nodes)
A (25%)
B (25%)
C (25%)
D (25%)
Node 1
os.type=ubuntu
os.version=14.10
glibc.version=2.20
JDK.version=8u20
Node 2
os.type=RHEL
os.version=5.1
GPU.type=x86_64
JDK.version=7u20
Node 16
os.type=windows
os.version=7
JDK.version=8u20
Node 17
os.type=SUSE
os.version=12
GPU.type=i686
JDK.version=7u20

Questions?

Apache Hadoop YARN: state of the union

More Related Content

What's hot (20)

Similar to Apache Hadoop YARN: state of the union (20)

More from DataWorks Summit (20)

Recently uploaded (20)

Apache Hadoop YARN: state of the union