YARN Services
Download as PPTX, PDF•4 likes•1,728 views
This document discusses YARN services in Hadoop, which allow long-lived applications to run within a Hadoop cluster. YARN (Yet Another Resource Negotiator) provides an operating system-like platform for data processing by allowing various applications to share cluster resources. The document outlines features for long-lived services in YARN, including log aggregation, Kerberos token renewal, and service registration/discovery. It also discusses how Hadoop 2.6 and later versions implement these features to enable long-running applications that can withstand failures.
![YARN-913 Service Registry
$ slider resolve --path ~/services/org-apache-slider/storm1
{ "type" : "JSONServiceRecord",
"external" : [ {
"api" : "http://",
"addressType" : "uri",
"protocolType" : "webui",
"addresses" : [ {
"uri" : "http://nn.example.com:46132"
} ]
}, {
"api" : "classpath:org.apache.slider.publisher.configurations",
"addressType" : "uri",
"protocolType" : "REST",
"addresses" : [ {
"uri" : "http://nn.example.com:46132/ws/v1/slider/publisher/slider"
}]
} } ] }](https://image.slidesharecdn.com/2014-11-16-services-final-141117092525-conversion-gate01/85/YARN-Services-16-320.jpg)
![Internal and external
"internal" : [ {
"api" : "classpath:org.apache.slider.agents.secure",
"addressType" : "uri",
"protocolType" : "REST",
"addresses" : [ {
"uri" : "https://nn.example.com:47749/ws/v1/slider/agents"
} ]
} ]](https://image.slidesharecdn.com/2014-11-16-services-final-141117092525-conversion-gate01/85/YARN-Services-17-320.jpg)
![$ yarn rmadmin
...
-addToClusterNodeLabels [label1,label2,label3]
-removeFromClusterNodeLabels [label1,label2,label3]
-replaceLabelsOnNode [node1:port,label1,label2]
-directlyAccessNodeLabelStore
Labels](https://image.slidesharecdn.com/2014-11-16-services-final-141117092525-conversion-gate01/85/YARN-Services-24-320.jpg)
YARN Services
- 1. Hadoop YARN Services Steve Loughran– Hortonworks stevel at hortonworks.com @steveloughran ApacheCon EU, November 2014
- 2. Apache Hadoop + YARN: An OS for data
- 3. An OS can do more than SQL statements
- 4. An OS can do more than run admin-installed apps
- 5. An OS lets you run whatever you want!
- 6. An OS Offers • Persistent Storage • Execution of code • jobs & services • scheduling • Communications • Security
- 7. YARN Services: Long lived applications within a Hadoop cluster
- 8. Background: YARN YARN Node Manager HDFS YARN Resource Manager “The RM” HDFS YARN Node Manager HDFS YARN Node Manager HDFS • Servers run YARN Node Managers (NM) • NM's heartbeat to Resource Manager (RM) • RM schedules work over cluster • RM allocates containers to apps • NMs start containers • NMs report container health
- 9. Client creates App Master YARN Node Manager HDFS YARN Resource Manager “The RM” HDFS YARN Node Manager HDFS YARN Node Manager HDFS Client Application Master
- 10. “AM” requests containers YARN Node Manager HDFS YARN Resource Manager HDFS YARN Node Manager HDFS YARN Node Manager Application Master HDFS Container Container Container
- 11. Short lived applications • failure: clean restart • logs: collect at end • placement: by data • security: Kerberos delegation tokens • discovery: launcher app can track
- 12. Long-lived services • failure: stay available • logs: ongoing collection • placement: availability, performance • security: ?? • discovery: ???
- 13. YARN-896 Support for YARN services:
- 14. YARN-896 Log aggregation Kerberos token renewal Gang scheduling Service registration & discovery Net & Disk resources Windowed failure tracking Container reuse Anti-affinity placement Container resource flexing Container signalling Labelled nodes & queues Applications to continue over AM restart REST
- 15. Hadoop 2.6 Log aggregation Kerberos token renewal Gang scheduling Service registration & discovery Net & Disk resources Windowed failure tracking Container reuse Anti-affinity placement Container resource flexing Container signalling Labelled nodes & queues Applications to continue over AM restart (Docker) REST
- 16. YARN-913 Service Registry $ slider resolve --path ~/services/org-apache-slider/storm1 { "type" : "JSONServiceRecord", "external" : [ { "api" : "http://", "addressType" : "uri", "protocolType" : "webui", "addresses" : [ { "uri" : "http://nn.example.com:46132" } ] }, { "api" : "classpath:org.apache.slider.publisher.configurations", "addressType" : "uri", "protocolType" : "REST", "addresses" : [ { "uri" : "http://nn.example.com:46132/ws/v1/slider/publisher/slider" }] } } ] }
- 17. Internal and external "internal" : [ { "api" : "classpath:org.apache.slider.agents.secure", "addressType" : "uri", "protocolType" : "REST", "addresses" : [ { "uri" : "https://nn.example.com:47749/ws/v1/slider/agents" } ] } ]
- 18. Failures YARN Node Manager HDFS YARN Resource Manager HDFS YARN Node Manager HDFS YARN Node Manager Application Master HDFS Container Container Container
- 19. Failures YARN Node Manager HDFS YARN Resource Manager HDFS YARN Node Manager HDFS Container Container
- 20. Failures YARN Node Manager HDFS YARN Resource Manager HDFS YARN Node Manager HDFS Application Master Container Container container 1 container 2 lost: container 3
- 21. Easy: enabling // Client amLauncher.setKeepContainersOverRestarts(true); amLauncher.setMaxAppAttempts(8); // Server List<Container> liveContainers = amRegistrationData.getContainersFromPreviousAttempts();
- 22. Harder: rebuilding state Persisted Rebuilt Transient Node Map Specification Placement History Event History Component Map Container Queues
- 23. Log Aggregation <property> <name>yarn.log-aggregation-enable</name> <value>true</value> </property>
- 24. $ yarn rmadmin ... -addToClusterNodeLabels [label1,label2,label3] -removeFromClusterNodeLabels [label1,label2,label3] -replaceLabelsOnNode [node1:port,label1,label2] -directlyAccessNodeLabelStore Labels
- 25. Labels offer • Separation of workloads • Separation of service roles • Separation of production & dev code • Allocation to specific hardware classes
- 26. Security • Token expiry a core Kerberos feature • Token expiry inimical to service longevity • Specifically: token delegation
- 27. Security YARN: AM/RM token renewal NM HDFS access for AM container relaunch You: embrace keytabs, test lots
- 28. …so you can now • Write long lived apps • with failure resilience • centralised log viewing • labelled/isolated placement • in secure clusters
- 29. Why not just use Mesos?
- 31. Hadoop 2.7+ Log aggregation Kerberos token renewal Gang scheduling Service registration & discovery Docker Net & Disk resources Windowed failure tracking Container reuse Anti-affinity placement Container resource flexing Container signalling Labelled nodes & queues Applications to continue over AM restart REST