On-premise Spark as a Service with YARN
Download as PPTX, PDF•0 likes•190 views
On Premise Spark-as-a-Service on YARN provides Spark-as-a-service in Sweden using Hopsworks, which was built on Hops Hadoop and provides multi-tenant Spark/Kafka/Flink jobs as a service. Hopsworks uses X.509 certificates for authentication instead of Kerberos and provides project-based access control and quotas. It simplifies writing secure Spark Streaming applications with Kafka.
![Livy to launch Spark 2.0 Jobs
[Image from: http://gethue.com]](https://image.slidesharecdn.com/spark-summit-europe-oct16-dowling-spark-as-a-service-sics-161128135854/85/On-premise-Spark-as-a-Service-with-YARN-17-320.jpg)
![Summary
• Hopsworks provides first-class support for
Spark-as-a-Service
– Streaming or Batch Jobs
– Zeppelin Notebooks
• Hopworks simplifies writing secure
SparkStreaming applications with Kafka
21http://github.com/hopshadoop
Hops
[Hadoop For Humans]](https://image.slidesharecdn.com/spark-summit-europe-oct16-dowling-spark-as-a-service-sics-161128135854/85/On-premise-Spark-as-a-Service-with-YARN-21-320.jpg)
On-premise Spark as a Service with YARN
- 1. On Premise Spark-as-a-Service on YARN Jim Dowling Associate Prof @ KTH, Stockholm Senior Researcher, SICS Swedish ICT CEO, Logical Clocks AB Twitter: @jim_dowling
- 2. Spark-as-a-Service in Sweden • SICS ICE: datacenter research and test environment • Hopsworks: Spark/Kafka/Flink/Hadoop-as-a-service – Built on Hops Hadoop (www.hops.io) – Over 100 active users – Spark the platform of choice 2
- 4. Hops-YARN Architecture 4 ResourceMgrs NDB Scheduler YARN Client NodeManagers Resource Trackers Heartbeats (70-95%) AM Reqs (5-30%)
- 5. Pluggable DB: Data Abstraction Layer 5 NameNode (Apache v2) DAL API (Apache v2) NDB-DAL-Impl (GPL v2) Other DB (Other License) hops-2.7.3.jar dal-ndb-2.7.3-7.5.4.jar
- 6. 6 HopsFS Throughput vs Apache HDFS NDB Setup: Nodes using Xeon E5-2620 2.40GHz Processors and 10GbE. NameNodes: Xeon E5-2620 2.40GHz Processors machines and 10GbE.
- 7. HOPSWORKS 7
- 8. Project-Based Multi-Tenancy • A project is a collection of – Users with Roles – HDFS DataSets – Kafka Topics – Notebooks, Jobs • Per-Project quotas – Storage in HDFS – CPU in YARN • Uber-style Pricing • Sharing across Projects – Datasets/Topics 8 project dataset 1 dataset N Topic 1 Topic N Kafka HDFS
- 10. Look Ma, No Kerberos! • For each project, a user is issued with a X.509 certificate, containing the project-specific userID. • Inspired by Netflix’ BLESS system. • Services are also issued with X.509 certificates. – Both user and service certs are signed with the same CA. – Services extract the userID from RPCs to identify the caller.
- 12. 12 [email protected] 1. Launch Spark Job Distributed Database 2. Get certs, service endpoints YARN Private LocalResources Spark Streaming App 4. Materialize certs 3. YARN Job, config 6. Get Schema 7. Consume Produce 5. Read Certs Hopsworks KafkaUtil Spark Streaming on YARN with Hopsworks 8. Authenticate
- 13. Spark Stream Producer in Secure Kafka SparkConf sparkConf = … JavaSparkContext jsc = … 1. Discover: Schema Registry and Kafka Broker Endpoints 2. Create: Kafka Properties file with certs and broker details 3. Create: producer using Kafka Properties 4. Download: the Schema for the Topic from the Schema Registry 5. Distribute: X.509 certs to all hosts on the cluster 6. Cleanup securely // write to Kafka 13 Developer Operations
- 14. Spark Streaming Producer in Hopsworks List<String> topics = KafkaUtil.getTopics(); … SparkProducer sparkProducer = KafkaUtil.getSparkProducer(topic); … Map<String, String> message = … sparkProducer.produce(message); … sparkProducer.close(); 14https://github.com/hopshadoop/hops-kafka-examples
- 15. Spark Streaming Consumer in Hopsworks JavaStreamingContext jssc = … List<String> topics = KafkaUtil.getTopics(); … SparkConsumer consumer = KafkaUtil.getSparkConsumer(jssc, topics); … // Avro schema downloaded by framework here GenericRecord genericRecord = KafaUtil.getRecordInjections() .get(topic); … jssc.start(); jssc.awaitTermination(); 15 https://github.com/hopshadoop/hops-kafka-examples
- 16. Zeppelin Support for Spark/Livy 16
- 17. Livy to launch Spark 2.0 Jobs [Image from: http://gethue.com]
- 18. Debugging Spark with DrElephant • Project-specific view of performance/correctness issues for completed Spark Jobs • Customizable heuristics • Doesn’t show killed jobs
- 19. Karamel/Chef for Automated Installation 19 Google Compute Engine BareMetal
- 20. 20 Demo
- 21. Summary • Hopsworks provides first-class support for Spark-as-a-Service – Streaming or Batch Jobs – Zeppelin Notebooks • Hopworks simplifies writing secure SparkStreaming applications with Kafka 21http://github.com/hopshadoop Hops [Hadoop For Humans]
- 22. Hops Team Active: Jim Dowling, Seif Haridi, Tor Björn Minde, Gautier Berthou, Salman Niazi, Mahmoud Ismail, Theofilos Kakantousis, Konstantin Popov, Antonios Kouzoupis, Ermias Gebremeskel. Alumni: Vasileios Giannokostas, Johan Svedlund Nordström, Rizvi Hasan, Paul Mälzer, Bram Leenders, Juan Roca, Misganu Dessalegn, K “Sri” Srijeyanthan, Jude D’Souza, Alberto Lorente, Andre Moré, Ali Gholami, Davis Jaunzems, Stig Viaene, Hooman Peiro, Evangelos Savvidis, Steffen Grohsschmiedt, Qi Qi, Gayana Chandrasekara, Nikolaos Stanogias, Ioannis Kerkinos, Peter Buechler, Pushparaj Motamari, Hamid Afzali, Wasif Malik, Lalith Suresh, Mariano Valles, Ying Lieu.
Editor's Notes
- #8: Nobody gets a cluster ….. everybody gets a project!
- #9: Privileges – upload/download data, run analysis jobs Like RBAC solution. All access via HopsWorks.
- #10: 9
- #12: 11
- #13: 12
- #14: public class HopsKafkaUtil implements Serializable { KAFKA_BROKERADDR_ENV_VAR = "kafka.brokeraddress"; KAFKA_RESTENDPOINT = "kafka.restendpoint"; KAFKA_SESSIONID_ENV_VAR = "kafka.sessionid"; KAFKA_PROJECTID_ENV_VAR = "kafka.projectid"; KAFKA_K_CERTIFICATE_ENV_VAR = "kafka_k_certificate"; KAFKA_T_CERTIFICATE_ENV_VAR = "kafka_t_certificate"; String getHopsConsumer(String topic) {…} String getHopsProducer(String topic) {…} String getHopsSparkKafkaConsumer(String topic) {…} String getHopsSparkKafkaProducer(String topic) {…} String getSchema(String topicName, int versionId) {..} Map<String, String> getKafkaProps(String propsStr) {…} }
- #15: HopsKafkaProperties.defaultProps())
- #16: HopsKafkaProperties.defaultProps())
- #17: https://gist.github.com/rawkintrevo/ad206879753733f5a536
- #19: Netty dependency conflict with our app in blocking mode Impacts: application size, main class run on our multi-tenant application - System.exit(), logs are written locally No accumulator results or exceptions from the ExecutionEnvironment.execute() call Can only kill YARN job, not Spark session – cleanup issues Spark Dispatcher The client directly starts the Job in YARN, rather than bootstrapping a cluster and after that submitting the job to that cluster. The client can hence disconnect immediately after the job was submitted All user code libraries and config files are directly in the Application Classpath, rather than in the dynamic user code class loader Containers are requested as needed and will be released when not used any more The “as needed” allocation of containers allows for different profiles of containers (CPU / memory) to be used for different operators