Powering Predictive Mapping at Scale with Spark, Kafka, and Elastic Search: Spark Summit East talk by Jorg Schad
7 likes1,350 views
The document discusses powering predictive mapping at scale using the SMACK stack, which includes Spark, Kafka, and Elasticsearch. It describes how the SMACK stack can ingest millions of events per second from connected devices, store the data in Apache Spark, and allow real-time and batch processing of the data. It also provides an example of using the stack for real-time tracking of geo-enabled IoT devices and demonstrates the data flow and a demo of the system.
Powering Predictive Mapping at Scale with Spark, Kafka, and Elastic Search: Spark Summit East talk by Jorg Schad
- 1. Š 2016 Mesosphere, Inc. All Rights Reserved. 1 @joerg_schad @dcos #smack Powering Predictive Mapping at Scale with Spark, Kafka, and Elastic Search Spark Summit East February 08, 2017
- 2. Š 2016 Mesosphere, Inc. All Rights Reserved. 2 JÜrg Schad Distributed Systems Engineer @joerg_schad
- 3. Š 2016 Mesosphere, Inc. All Rights Reserved. 3 HYPERSCALE MEANS VOLUME AND VELOCITY Batch Event ProcessingMicro-Batch Days Hours Minutes Seconds Microseconds Solves problems using predictive and prescriptive analyticsReports what has happened using descriptive analytics Predictive User InterfaceReal-time Pricing and Routing Real-time AdvertisingBilling, Chargeback Product Recommendations
- 4. Š 2016 Mesosphere, Inc. All Rights Reserved. 4 SMACK stack EVENTS Ubiquitous data streams from connected devices INGEST Apache Kafka STORE Apache Spark ANALYZE Apache Cassandra ACT Akka Ingest millions of events per second Distributed & highly scalable database Real-time and batch process data Visualize data and build data driven applications DC/OS Sensors Devices Clients
- 5. Š 2016 Mesosphere, Inc. All Rights Reserved. 5 NAIVE APPROACH Typical Datacenter siloed, over-provisioned servers, low utilization Industry Average 12-15% utilization mySQL microservice Cassandra Spark/Hadoop Kafka
- 6. Š 2016 Mesosphere, Inc. All Rights Reserved. 6 Mesos & DC/OS
- 7. Š 2016 Mesosphere, Inc. All Rights Reserved. 7 MULTIPLEXING OF DATA, SERVICES, USERS, ENVIRONMENTS Typical Datacenter siloed, over-provisioned servers, low utilization Mesos/ DC/OS automated schedulers, workload multiplexing onto the same machines mySQL microservice Cassandra Spark/Hadoop Kafka
- 8. Š 2016 Mesosphere, Inc. All Rights Reserved. 8 DC/OS ENABLES MODERN DISTRIBUTED APPS Datacenter Operating System (DC/OS) Distributed Systems Kernel (Mesos) Big Data + Analytics EnginesMicroservices (in containers) Streaming Batch Machine Learning Analytics Functions & Logic Search Time Series SQL / NoSQL Databases Modern App Components Distributed systems kernel to abstract resources Ecosystem of frameworks & apps Consistent architecture to run on top of kernel User Interface (GUI & CLI) Core system services (e.g., distributed init, cron, service discovery, package mgt & installer, storage) Any Infrastructure (Physical, Virtual, Cloud)
- 9. Š 2016 Mesosphere, Inc. All Rights Reserved. 9 EXAMPLE: REAL-TIME TRACKING
- 10. Š 2016 Mesosphere, Inc. All Rights Reserved. 10 GEO-ENABLED IoT
- 11. Š 2016 Mesosphere, Inc. All Rights Reserved. 11 DATA FLOW
- 12. Š 2016 Mesosphere, Inc. All Rights Reserved. 12 DEMO
- 13. Š 2016 Mesosphere, Inc. All Rights Reserved. 13 THANK YOU! ANY QUESTIONS? @dcos [email protected] /groups/8295652 /dcos /dcos/examples /dcos/demos chat.dcos.io
- 14. Š 2017 Mesosphere, Inc. All Rights Reserved. 14 Keep it running!
- 15. Š 2016 Mesosphere, Inc. All Rights Reserved. 15 SERVICE OPERATIONS â Configuration Updates (ex: Scaling, re-configuration) â Binary Upgrades â Cluster Maintenance (ex: Backup, Restore, Restart) â Monitor progress of operations â Debug any runtime blockages
- 16. Š 2016 Mesosphere, Inc. All Rights Reserved. 16 Typical Use: distributed, large-scale data processing; micro-batching Why Spark Streaming? â Micro-batching creates very low latency, which can be faster â Well defined role means it fits in well with other pieces of the pipeline APACHE SPARK (STREAMING)