Developing apache spark jobs in .net using mobius
- 1. Developing Apache Spark Jobs in .NET using Mobius Kaarthik Sivashanmugam @kaarthikss dotnetfringe 2016
- 2. Apache Spark • General purpose cluster computing system for big data processing and analytics • Ease of programming • High performance • Unified API to solve a diverse set of complex data problems • API in Scala, Java, Python & R
- 3. Apache Spark Key Concepts • Data • RDD – Resilient Distributed Dataset • Transformation & Action • DataFrame • Dstream • Cluster • Driver • Executor
- 4. Mobius: C# API for Spark • Enable organizations invested deeply in .NET to build Apache Spark applications in C# • Reuse of existing .NET libraries in Spark applications
- 5. .NET & Spark Scala/Java API SparkR PySpark Mobius: C# API Apache Spark Spark Apps in .NET
- 6. Word Count in Spark using RDD Scala RDD of lines in the file RDD of words in the file RDD of tuple - (word, 1) RDD of tuple - (word, count) Action that triggers job
- 7. Word Count in Spark using RDD C# Scala F#
- 8. Develop & Launch Mobius Applications Spark Client A Get Mobius release B Get Mobius driver and dependencies 1 Add Reference to Mobius package in NuGet 2 Develop, debug, test Mobius driver application 3 Build Mobius driver Run sparkclr-submit.cmd or sparkclr-submit.sh C Runs Spark job Example: sparkclr-submit.cmd --master spark://IP:PORT --total-executor-cores 200 --executor-memory 12g -- conf spark.eventLog.enabled=true -- conf spark.eventLog.dir=hdfs://nn/path/to/eventlog --exe Pi.exe D:MobiusexamplesPi
- 9. Demo Implementing a simple Mobius driver program using DataFrame
- 10. Structured Data in Mobius using DataFrame JSON Cassandra Note – Dataset is replacing DataFrame in Spark
- 11. Mobius & Spark C# Worker CLR IPC Sockets C# Worker CLR IPC Sockets C# Worker CLR IPC Sockets C# Driver CLR IPC Sockets SparkExecutor SparkExecutor SparkExecutor SparkContext JVM JVM JVM JVM Workers Driver Mobius can be used with any existing Spark cluster (Standalone, YARN) in Windows & Linux
- 12. Mobius in Linux • Mono is used for using Mobius with Spark in Linux • Mobius project CI (build, unit & functional tests) in Ubuntu • Mobius validated in Ubuntu, CentOS, OSX • Mobius validated with Spark clusters in Azure HDInsight and Amazon Web Services EMR • More info at linux-instructions.md @ GitHub
- 13. Kafka Message Processing in Mobius using DStream Initialize StreamingContext & Checkpoint Create Kafka DStream Use DStream transformations to count logs by loglevel within a time window Save log count Start stream processing
- 14. Internals of Driver & Worker
- 15. Driver-side Interop CSharpRunner JVM 1 Launch sparkclr-submit.cmd or sparkclr-submit.sh CSharpBackendLaunch Netty server creating proxy for JVM calls 2 C# Driver Launch C# process using port number from CSharpBackend 3 CLR SparkConf SparkContext Create and manage Proxies for JVM objects SparkConf SparkContext Interop Components Mirror C#-side operations Invoke JVM methods RDD DataFrame DStream …CSharpRDD RDD DataFrame DStream PipelinedRDD …
- 16. 1 Compute 2 CLR CSharpWorker.exe Launch Worker-side Interop JVM CSharpRDD Executor Spark Worker 3 Read bytes 5 Write bytes 4 Execute C# operation 1 Compute
- 17. Mobius Project Info • https://github.com/Microsoft/Mobius • MIT license • Discussions • StackOverflow: tag “SparkCLR” • Gitter: https://gitter.im/Microsoft/Mobius • Twitter: @MobiusForSpark
- 18. Mobius Project Status • Past Releases • v1.5.200 (Spark 1.5.2) • v1.6.100 (Spark 1.6.1) • Upcoming Releases • V1.6.200 (Spark 1.6.2) • v2.0.000 (Spark 2.0.0) • Work planned/in progress • Support for interactive scenarios (Zeppelin/Jupyter integration – IfSharp?) • Exploration of support for ML scenarios • Idiomatic F# API (?) • Support for .NET Core
- 19. Thank you Mobius is production-ready & cloud-ready Use Mobius to build Apache Spark jobs in .NET Contribute to github.com/Microsoft/Mobius @MobiusForSpark
Editor's Notes
- #3: RDD – fault tolerant collection of elements partitioned across the nodes of the cluster that can be operated on in parallel Persist an RDD in memory, allowing it to be reused efficiently across parallel operations RDDs automatically recover from node failures. Transformations, create a new dataset from an existing one – transformations are lazy actions, which return a value to the driver program after running a computation on the dataset. DataFrame is a distributed collection of data organized into named columns Dstream - represents a continuous stream of data.
- #4: RDD – fault tolerant collection of elements partitioned across the nodes of the cluster that can be operated on in parallel Persist an RDD in memory, allowing it to be reused efficiently across parallel operations RDDs automatically recover from node failures. Transformations, create a new dataset from an existing one – transformations are lazy actions, which return a value to the driver program after running a computation on the dataset. DataFrame is a distributed collection of data organized into named columns Dstream - represents a continuous stream of data.
- #9: Exe icon credit – Icon made by Freepik from www.flaticon.com