Authoring and Hosting Applications on YARN using Slider

© Hortonworks Inc. 2011 – 2015. All Rights Reserved
Authoring and Hosting Applications on
YARN using Slider
June 2015
Sumit Mohanty
Jonathan Maron

Agenda
Background
Simplified Packaging
Application Upgrade
Security Enhancements
Docker Integration
Key Enhancements
Conclusion and Q/A
Page 2

Agenda
Background
Application Upgrade
Docker Integration
Key Enhancements
Conclusion and Q/A
Page 3

Fun operational problems
Placement: where to run?
Installation
Configuration & Binding
Client configuration
Lifecycle
Failure handing and recovery
Logging
Upgrading
Metrics & Monitoring
Start/Stop
Reconfigure
Scale up/down
Rolling-restart
Decommission/Recommission

== Getting your code to work across
a set of fairly reliable machines

Apache Slider
Deploying and managing
applications on a YARN cluster
http://slider.incubator.apache.org/

YARN runs code across the cluster
Page 7
HDFS
YARN Node Manager
HDFS
YARN Node Manager
HDFS
YARN Resource Manager
“The RM”
HDFS
YARN Node Manager
• Servers run YARN Node Managers
• NM's heartbeat to Resource Manager
• RM schedules work over cluster
• RM allocates containers to apps
• NMs start containers
• NMs report container health

Client creates App Master
Page 8
HDFS
YARN Node Manager
HDFS
YARN Node Manager
HDFS
“The RM”
HDFS
YARN Node Manager
Client
Application Master

AM asks for containers
Page 9
HDFS
YARN Node Manager
HDFS
YARN Node Manager
HDFS
HDFS
YARN Node Manager
Application Master
Container
Container
Container

Slider view of an Application
An application is a set of components
A component is a daemon/launched exe
configuration
scripts, data files, etc.
Component may have one or more instances
Component instances are managed
By extension, the app instance is
Example
HBase Application (3 components)
• HBase Master
• HBase RegionServer
• HBase REST service
Page 10

Agenda
Background
Application Upgrade
Docker Integration
Key Enhancements
Conclusion and Q/A
Page 11

Options
A Simple Program
Simple Program that Advertises
Bring your own binaries
A Minimal Application Package
Page 12
Goal: Make it simpler to run simple apps

A Simple Program
All dependencies installed
Default configuration
Well-known endpoints
Long running
Page 13

Simple Program that Advertises
All dependencies installed
Portable – advertises the endpoints
Long running
Page 16

Bring your own Binaries
Dependencies deployed by the app instance
Portable – advertises the endpoints
Long running
Page 20

Demo
Simplified Application Package

Application Package
Why might I need a package?
Easier to share
Versioned
Default configs
Easier to upgrade
• admins may install blessed versions
May package a client
• slider client install --package … --client-config …
Page 24
HBase Accumulo Storm Kafka Solr Spark …

Minimal Application Package
Package content
Page 25
Archive: jmemcached-1.0.0.zip
Length Date Time Name
--------- ---------- ----- ----
472 02-26-2015 02:49 appConfig-default.json
2029 11-23-2014 14:11 metainfo.xml
0 08-27-2014 20:54 package/
0 01-09-2015 01:34 package/files/
118272 01-09-2015 01:33 package/files/jmemcached-1.0.0.tar
0 11-23-2014 14:11 package/scripts/
1659 11-23-2014 14:11 package/scripts/memcached.py
1346 11-23-2014 14:11 package/scripts/params.py
1823 11-23-2014 14:11 README.txt
280 11-23-2014 14:11 resources-default.json
1100 02-26-2015 02:59 metainfo.json
--------- -------
126981 11 files

Agenda
Background
Application Upgrade
Docker Integration
Key Enhancements
Conclusion and Q/A
Page 26

YARN Containers with Slider (recap)
Page 27
YARN Node Manager
Component (container)AppMaster (container)
YARN Node Manager
HDFS
Slider Agent
Application
Slider AppMaster
Slider Client
HDFS
HDFS

Upgrade Considerations
YARN Core
Slider apps run uninterrupted
Slider Client
Backwards compatible
Slider App Master
Applications started prior to YARN core upgrade/downgrade continue to run with same core
libraries
App Master upgrade requires app restart with newer client
Page 28

Slider Application Upgrades
Slider supports application rolling upgrade
Upgrade to a newer version of application binary and/or configuration, without downtime
Application owner orchestrates the upgrade
Sufficient “slaves” should be running to handle the load at the time of the upgrade
Ensure the application is compatible in a mixed version mode
Validate application health at important touch points
Perform downgrade at the earliest detection of anomaly
App Author
Pre-upgrade hook for simple housekeeping tasks (optional)
Page 29

Application Upgrade Steps
Page 30
Create an app:
$ slider create hbase_demo --resources resources_v1.0.json -–template appConfig_v1.0.json
$ slider list hbase_demo --containers
Provide Upgrade spec:
$ slider upgrade hbase_demo --resources resources_v2.0.json --template appConfig_v2.0.json
Upgrade components/containers:
$ slider upgrade hbase_demo --components HBASE_MASTER
$ slider upgrade hbase_demo --containers container_000002

Demo
Rolling Upgrade

Minimal Upgrade
diff appConfig_v1.0.json appConfig_v2.0.json
Page 32
< "application.def": ".slider/package/HBASE/0.98.4.2.2.0.0-2041/slider-hbase-app-package-0.98.4.2.2.0.0-2041-
hadoop2.zip",
---
> "application.def": ".slider/package/HBASE/1.1.0.2.3.0.0-2245/slider-hbase-app-package-1.1.0.2.3.0.0-2245.zip",
< "site.global.app_version": "0.98.4.2.2.0.0-2041",
< "site.global.app_root": "${AGENT_WORK_ROOT}/app/install/hbase-0.98.4.2.2.0.0-2041-hadoop2",
---
> "site.global.app_version": "1.1.0.2.3.0.0-2245",
> "site.global.app_root": "${AGENT_WORK_ROOT}/app/install/hbase-1.1.0.2.3.0.0-2245",

Agenda
Background
Application Upgrade
Docker Integration
Key Enhancements
Conclusion and Q/A
Page 33

Security
Applications validated to work in Kerberos secured cluster
Secure application created and keytabs available to application components
Security parameters specified in application configuration
User obtains TGT (kinit) prior to Slider application creation
E.g. HBase 0.98.4
Agent-AM SSL communication
One-Way by default
Two-Way can be enabled
Page 34

New Security Features
Application Keytabs
Slider distribution
• Keytabs installed into pre-defined location in HDFS can be localized by Slider
Manual distribution
Application certificate store creation and distribution
Application specifies need for keystore and/or truststore
Generated stores localized to standard locations on container hosts
Page 35

Agenda
Background
Application Upgrade
Docker Integration
Key Enhancements
Conclusion and Q/A
Page 36

Slider: current architecture
1. CLI
2. AppMaster
3. AgentProvider
4. SliderAgent
5. Docker containers
Slider
App config
files
“The RM”
YARN Node Manager
Slider Agent
YARN Node Manager
Slider Agent
Application Docker container
Slider Application Master
Slider CLI
Docker daemon
Docker daemon
Application Docker container

On-boarding Dockerized Application on YARN via Slider
• Application definition – metainfo.json (contains docker image name)
• Application instance configuration (if any) – app_config.json
• Application resource definition – resources.json
No Application bundle and lifecycle scripts
Page 38

metainfo.json – structure of the application
{
"schemaVersion": "2.1",
"application": {
"name": "NODEJS-REDIS",
"components": [
{
"name": "NODEJS",
"type": "docker",
"dockerContainers": [
{
"name": "nodejs”,
"image": "rsahahw/centos-node-redis”,
"ports": [{"containerPort" : "8000” }]
}
]
},
{
"name": "REDIS",
"type": "docker",
"dockerContainers": [
{
"name": "redis",
"image": "tutum/redis",
"ports": [{
"containerPort" : "6379",
"hostPort": "6379”
...

resources.json – resource requirement of the application
{
"schema" : "http://example.org/specification/v2.0.0",
"metadata" : {
},
"global" : {
},
"components": {
"NODEJS": {
"yarn.role.priority": "1",
"yarn.component.instances": "1",
"yarn.memory": "512"
},
"REDIS": {
"yarn.role.priority": "2",
"yarn.component.instances": "1",
"yarn.memory": "512"
}
}
}

app_config.json – instance specific configuration of the application
{
"schema": "http://example.org/specification/v2.0.0",
"metadata": {
},
"global": {
},
"components": {
"NODEJS": {
"nodejs.options":"-d -e REDIS_HOST=${REDIS_HOST}",
"nodejs.statusCommand":
"/usr/bin/docker exec ${CONTAINER_ID} ps aux | grep redis-server"
},
"REDIS": {
"redis.options":"-d -e REDIS_PASS=**None**",
"redis.statusCommand":
"docker exec ${CONTAINER_ID} ps aux | grep redis-server"
}
}
}

Commands: create, start, stop, destroy
$ slider create dockerapp1 --resources resources.json -–template appConfig.json -–metainfo
metainfo.json
$ slider list
$ slider stop dockerapp1
$ slider destroy dockerapp1

Demo
Dockerized node.js app using dockerized redis for persistence

Agenda
Background
Application Upgrade
Docker Integration
Key Enhancements
Conclusion and Q/A
Page 44

 Labelled nodes & queues
 Log aggregation
 Container retention over Application Master restart
 Service registration & discovery
 Kerberos token renewal
 Windowed Application failure tracking
YARN Enhancements (Hadoop 2.6+)
—essential for all long-lived YARN services

Service Registration and Discovery (YARN-913)
Core part of Hadoop 2.6
Registration & discovery of static & dynamic services
Slider
Find and communicate with AM (client)
AM-agent binding
Publication of REST/IPC endpoints for deployed apps
Configuration files
Page 46

Node Labels
Leveraged by slider to map components to labels
Page 47

Placement/Escalation
Long lived processes
May be willing to wait for an extended period to come up on the same node
• E.g. if a server has crashed and is rebooting, or is down for a quick hardware fix
Ultimately giving up is ok
• Can recover
Unreliability measured by failures in a recent time period
Page 48

Placement/Escalation Approach
‘Escalation’ - allow for extended periods to wait before relaxing placement
Minutes rather than seconds
Role & node failures designate, respectively, if an app is failing or a node is unreliable
Reset on a regular, schedule basis (tunable)
Node reliability tracking
Node failure counts do not include
• node-independent failures
– E.g. container memory limits exceeded
• pre-emption events
Component role failure counters don't count node failures or pre-emption
Untrusted nodes not asked for containers even if that node is the previous location container ran
Exception: If placement policy is designated as ‘strict’ there is no escalation
Page 49

Join in: Bring your favorite Applications to YARN
Page 50

Everyone is welcome
Useful Links
Website
• http://slider.incubator.apache.org/
Dev Mailing Lists
• dev@slider.incubator.apache.org
JIRA
• https://issues.apache.org/jira/browse/SLIDER
Page 51

Q/A
http://slider.incubator.apache.org/
Page 52

Thank you.
Page 53

Authoring and Hosting Applications on YARN using Slider

More Related Content

What's hot (20)

Viewers also liked (15)

Similar to Authoring and Hosting Applications on YARN using Slider (20)

More from DataWorks Summit (20)

Recently uploaded (20)

Authoring and Hosting Applications on YARN using Slider