Downloaded 41 times
Uploaded byAli Hodroj
Hybrid Transactional/Analytics Processing with Spark and IMDGs
This document discusses hybrid transactional/analytical processing (HTAP) with Apache Spark and in-memory data grids. It begins by introducing the speaker and GigaSpaces. It then discusses how modern applications require both online transaction processing and real-time operational intelligence. The document presents examples from retail and IoT and the goals of minimizing latency while maximizing data analytics locality. It provides an overview of in-memory computing options and describes how GigaSpaces uses an in-memory data grid combined with Spark to achieve HTAP. The document includes deployment diagrams and discusses data grid RDDs and pushing predicates to the data grid. It describes how this was productized as InsightEdge and provides additional innovations and reference architectures.
More Related Content
![[Webinar] Measure Twice, Build Once: Real-Time Predictive Analytics](https://cdn.slidesharecdn.com/ss_thumbnails/infochimpsthinkbigwebinar-130510142153-phpapp02-thumbnail.jpg?width=640&height=640&fit=bounds)
Similar to Hybrid Transactional/Analytics Processing with Spark and IMDGs
Hybrid Transactional/Analytics Processing with Spark and IMDGs
- 1. 1 Hybrid Transactional/Analytics Processing withSpark and In-Memory Data Grids Copyright © GigaSpaces 2016. All rights reserved. Ali Hodroj VP, Products and Strategy
- 2. 2 About me • VicePresident, Products and Strategy @ GigaSpaces • (ex) Director of Solutions Architecture • Blogging at http://blog.gigaspaces.com • @ahodroj • Email: [email protected] • Slides at http://slideshare.com/ahodroj
- 3. 3 About GigaSpaces Direct customers 500+ Headquarters NewYork, NY Established 2001
- 4. 4 Do we needto bridge online transaction processing with real-time operational intelligence?
- 5. 5 Modern applications: theline is blurred between… Transactional Analytical Essential to operate the business Turning data into value: insights, diagnosis, decision making &
- 6. 6 Stories from thereal #enterprise world...
- 7.
- 8.
- 9.
- 10. 10 Minimize Latency + StrongConsistency Maximize Data- Analytics Locality Goal:
- 11. 11 There’s a namefor it...
- 12.
- 13.
- 14. In-Memory Computing 101 DistributeCache Partitioned cache nodes In-Memory Data Grid Scale-out system of record Increased Capacity No support for write-heavy scenarios Limited to ID-based reads Reads are the only part latency path In-Memory Database Scale-up system of record
- 15. Heavy Read/Write –sharded/partitioned architecture Horizontally scalable on commodity HW (or cloud) Serves as system of record with querying & transaction semantics Requires modifying your application’s data access layer Distribute Cache Partitioned cache nodes In-Memory Data Grid Scale-out system of record In-Memory Database Scale-up system of record In-Memory Computing 101
- 16. Read/Write Scalability Drop-in SQLdatabase replacement Often lacks horizontal scalability (Joins) Requires replacing your database Distribute Cache Partitioned cache nodes In-Memory Data Grid Scale-out system of record In-Memory Database Scale-up system of record In-Memory Computing 101
- 17. IMDG Data Models @SpaceClass publicclass Product { private String name; private String brand; private Integer quantity; // … }
- 18. IMDG Data Placement– Fixed Hashing hash(key) % #nodes
- 19. IMDG Fixed Hashing- HA hash(key) % #nodes
- 20. 20 In-Memory Data Grids:How it works http://xap.github.io
- 21. 21 The database goes tothe background Partition your data and store it in memory In-Memory Data Grids: How it works http://xap.github.io
- 22. 22 Partitioned, co- located in-memory messaging In-MemoryData Grids: How it works http://xap.github.io
- 23. 23 Business logic, data& messaging co-located & partitioned into processing units In-Memory Data Grids: How it works http://xap.github.io
- 24. 24 Hot backup foreach partition for high availability In-Memory Data Grids: How it works http://xap.github.io
- 25. 25 Host your web applicationon the XAP infrastructure In-Memory Data Grids: How it works http://xap.github.io
- 26. 26 Auto-scale out &in based on real-time performance & load In-Memory Data Grids: How it works http://xap.github.io
- 27. 27 In-Memory Data Grids:How it works http://xap.github.io
- 28. 28 Host Cisco UCSServer CPU Intel 16core 2.9GHz Concurrent Threads 2 Throughput 200, 400, 800 ops/sec
- 29. 29 WHAT’S THE RIGHT DATA STORETO CHOOSE?
- 30. 30 ● Nope: Yourdata sources and applications are often distributed. ● In-Memory or not, these databases aren’t built for horizontal scale-out Approach Challenge Just an IMDB Thing…. Shove it all in one “Big Iron”?
- 31. 31 ● Not whenyour apps requires polyglot analytics ● Unless you want to write ML algorithms, MDX engines…etc from scratch Approach Challenge One large In- Memory Data Grid to Rule them all?
- 32. 32 What we needed Low-latencyScale-Out In- Memory Data Grid Large-scale distributed analytics framework Maximize Data- Analytics Locality Minimize Application Latency
- 33. 33 Our approach toHTAP Low-latency Scale-Out In-Memory Data Grid Large-scale distributed analytics framework +
- 34. 34 SPARK? So why didwe bet on
- 35. 35 • Unified &Concise API • Highly Flexible Data Store Integration • Massive Community and Adoption
- 36.
- 37. 37 Spark is cachingover <insert your data store>, not an in-memory system of record
- 38. 38 APACHE SPARK FIT INTOTHIS? How does
- 39. 39 In-Memory Data Grid In-MemoryStore(RAM) Flash, SSD, Off-Heap Store Spark Spark SQL Spark Steaming Machine Learning Highavailability Security&Management InsightEdge Core Building out the driver Transactional Tier ACID-compliant Strong Consistency Analytics Tier
- 40. 40 Data Grid +Spark Deployment Layout node 1 Spark master Grid master node 2 Spark worker Grid worker node 3 Spark worker Grid worker
- 41. 41 • List ofparent RDDs – Empty • An array of partitions that a dataset is divided to – IMDG Distributed Query to get partitions and their hosts • A compute function to do a computation on partitions – Iterator over portion of data • Optional preferred locations, i.e. hosts for a partition where the data will be loaded – hosts from Distributed Query Data Grid RDD: resilient distributed dataset
- 42. 42 node 1 Spark executor DataGrid RDD: one-to-one partition Spark Partition #1 Grid Partition #1 Direct connection Simple, but not enough parallelism for Spark node 2 Spark executor Spark Partition #2 Grid Partition #2 node 3 Spark executor Spark Partition #3 Grid Partition #3
- 43. 43 node 1 Spark Executor GridPrimary #1 Data Grid RDD: with bucketing 0 . . 1 . . 2 . . 3 . . 4 . . 5 . . . . . . . . . . . . Spark Partition #1 1023 1 Spark partition = M grid buckets 1 Grid partition = N Spark partitions Spark Partition #2 Spark Partition #1
- 44. 44 Grid DataFrames: predicatespushdown & columns pruning Aggregation in Spark Filtering and columns pruning in Data Grid SELECT SUM(amount) FROM order WHERE city = ‘NY’ AND year > 2012 Spark SQL architecture: • Pushing down predicates to Data Grid • Leveraging indexes • Transparent to user • Enabling support for other languages - Python/R Implementing DataSource API
- 45. 45 Push-down Predicates performance Traditional Spark filteringof 7MM records Grid-side + Spark filtering of 7MM records 31 seconds 1 second vs
- 46. 46 Eventually, we productizedthis as an open source Spark distribution
- 47. @InsightEdgeIO http://insightedge.io Apache 2License http://insightedge.io/docs http://insightedge.io/blog http://github.com/InsightEdge
- 48. GigaSpaces InsightEdge http://insightedge.io High PerformanceSpark with OLTP Capabilities
- 49.
- 50. 50 Spark GeoSpatial SQLand Data Frames
- 51. 51 • Multi-Spark Replication/ Federated Clusters In-Memory Replication across local or wide area networks
- 52. upcoming: Spark RDD/DFnative read/save on Off-Heap (SSD/Flash/Direct Buffers) Application Processi ng Primary instance s Backup instance s Sync Replicati on Storage Array Storage Array In Memory Data Grid Spark worker Spark worker • Significant RAM TCO reduction in Spark clusters • Direct RDD/DataFrame read write from SSD/Flash device • Avoid Filesystem hops and write amplification
- 53.
- 54. 5454 In-Process HTAP Read anyPOJO, JSON Document, or Transaction as a DataFrame or RDD Web services/apps can read any DataFrame as POJO True closed-loop analytics data pipeline @SpaceClass public class Product { private String name; private String brand; private Integer quantity; // … }
- 55. 5555 In-Memory Data Grid RealtimeReplication • Scoring models • Trigger actions • Events Transactions Analytics Point of Decision HTAP XAP + InsightEdge deployed on different grid clusters with bi- directional real-time data replication
- 56. 5656 Case Study: FleetGeo-analytics Challenge • Stream data from 1,000s of Taxis • Actively monitor and generate real-time notifications • Real-time Route Optimization and Geo-Fencing Solution • Leverage unified in-memory data fabric as middleware for geo-spatial analytics • Elastically scale stream processing and transactional apps together • Location-based tracking, Geo-fencing Edge components Data Sources
- 57.
- 58.