Bringing complex event processing to Spark streaming
Download as PPTX, PDF9 likes2,624 views
The document discusses Oracle's Continuous Query Processor (CQP) integrated with Apache Spark Streaming for complex event processing (CEP). It highlights CQP's functionalities such as event-by-event processing, pattern detection, and the benefits of a scalable and fault-tolerant architecture. The integration allows for real-time data analysis with efficient handling of events based on unique timestamps and flexible windowing.
Bringing complex event processing to Spark streaming
- 1. Copyright Š 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential â Internal/Restricted/Highly Restricted 1 Oracle Stream Analytics Complex Event Processing for Apache Spark Streaming Complex Event Processing for Spark Streaming Prabhu Thukkaram, Senior Director, Oracle Product Development Hoyong Park, Architect, Oracle Product Development
- 2. Copyright Š 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential â Internal/Restricted/Highly Restricted 2 Complex Event Processing with Continuous Query Processor Continuous Query Processor Pre-registered Queries E3@T3, E2@T2, E1@T1 ResultsInput Events R3@T3, R2@T2, R1@T1 Heartbeats
- 3. Copyright Š 2014 Oracle and/or its affiliates. All rights reserved. | Why Continuous Query Processor ? Oracle Confidential â Internal/Restricted/Highly Restricted 3 ⢠Complex event processing requires events to be processed one at a time â Each event must be processed as identified by its individual timestamp â Real world events originate at different times and must be processed as such â CEP applications seek correlation and patterns across events in time within or across batches and irrespective of batch boundaries ⢠Window length can span fractional batches ⢠Micro-batching with Spark Streaming â All events in the batch are identified by same time (RDD Time) â No progression of time between events in the same batch â No progression of time when RDD partitions are empty ⢠Critical for missing event scenarios. E.g. alert when order status âReceivedâ is not followed by âShippedâ for order Id 10001 within 1 hour
- 4. Copyright Š 2014 Oracle and/or its affiliates. All rights reserved. | Pattern Detection in CQP Oracle Confidential â Internal/Restricted/Highly Restricted 4 Checks if temperature readings from a power sensor are wobbling during a certain time interval. The CQP code checks for a W-Pattern in temperature readings during a 10 minute interval and selects support levels as output SELECT LAST(A.value), LAST(C.value) FROM TEMP_STREAM MATCH_RECOGNIZE ( PARTITION BY DEVICE_ID PATTERN (A+ B+ C+ D+) DURATION OF 10 MINUTES DEFINE A AS (value < PREV(value)) B AS (value > PREV(value)) C AS (value < PREV(value)) D AS (value > PREV(value)) ) A B C D 10 Minutes
- 5. Copyright Š 2014 Oracle and/or its affiliates. All rights reserved. | Distributed Complex Event Processing Oracle Confidential â Internal/Restricted/Highly Restricted 5 ⢠Continuous Query Processor â Event by event processing ⢠Each event is assigned a unique timestamp ⢠Apache Spark â Distributed computing with scale out and fault tolerance Spark Streaming + Continuous Query Processor = Distributed, Scalable, and Fault Tolerant CEP Platform
- 6. Copyright Š 2014 Oracle and/or its affiliates. All rights reserved. | How CQP complements Spark Streaming ? Oracle Confidential â Internal/Restricted/Highly Restricted 6 ⢠Continuous, event-by-event, and stateful processing ⢠Flexible temporal windows (Time & Rows) ⢠Automatic progression of time â Heartbeat propagation to advance time ⢠Pattern detection without batch boundaries â Built-in finite state automaton ⢠Declarative SQL like language
- 7. Copyright Š 2014 Oracle and/or its affiliates. All rights reserved. | How Spark complements CQP ? ⢠Distributed computing framework for CQP ⢠OOTB sources for data Ingestion ⢠Horizontal scale-out ⢠Fault tolerance and high availability ⢠Spark can detect failures, restart required resources, and automatically replay parts of the stream that experienced failure Oracle Confidential â Internal
- 8. Copyright Š 2014 Oracle and/or its affiliates. All rights reserved. | High-Level Architecture Data Stream Oracleâs CQP Engine Finite State Automaton for Pattern Detection across Discrete Events HDFS Journaled CQP engine State serialized to HDFS after computing each partition CQP Engine State restored on Executor Restart and Recompute of a partition Geo Sensing Cartridge for Spatial Analytics RETE Rules for Conditional Logic Complex Pattern Detection, Temporal Queries, Spatial Queries, Contextual Queries, and Conditional Logic are all executed in Oracleâs CEP Engine Distributed Cache for Ultra-fast Context Lookup
- 9. Copyright Š 2014 Oracle and/or its affiliates. All rights reserved. | CQP Spark Application Oracle Confidential â Internal Cluster Spark Standalone or YARN or Mesos Spark Driver Spark Executor Spark Executor Spark Executor ⢠Transformation delegated to CQP ⢠CQP application parsed in Sparkâs Driver process ⢠Driver builds a DAG (job definition) from CQP application ⢠Driver runs job for each micro-batch CQP CQP CQP
- 10. Copyright Š 2014 Oracle and/or its affiliates. All rights reserved. | Spark Integration - High Level ⢠Extensions â CQLDStream extends DStream overriding compute() â CQLRDD extends RDD overriding compute() ⢠Initialization/DAG creation phase â Create Spark Context, Streaming Context, and start CQP on each Executor â Setup DAG of CQLDStream ⢠Physical plan execution phase â Assign unique timestamp to each event in micro-batch based on RDD time â Register query with CQP on RDD compute if not already registered â Send micro-batch to CQP and accept returned results â Checkpoint CQP state if HA enabled Oracle Confidential â Internal
- 11. Copyright Š 2014 Oracle and/or its affiliates. All rights reserved. | Spark Integration - High Availability ⢠Executor Failure â Restart CQP on newly restarted Executor â On RDD compute, re-register query and restore query state from snapshots â Continue processing stream Oracle Confidential â Internal
- 12. Copyright Š 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential â Internal Demo
- 13. Copyright Š 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential â Internal Q & A
- 14. Copyright Š 2014 Oracle and/or its affiliates. All rights reserved. | CQP scalability in Spark Oracle Confidential â Internal/Restricted/Highly Restricted 14 0 20000 40000 60000 80000 w1(c3) w2(c5) w3(c7) w4(c9) Processing Time (seconds) for 40 Million Records 0 5000 10000 15000 20000 w1(c3) w2(c5) w3(c7) w4(c9) Avg. Processing Time Per Batch (milliseconds) 0 20 40 60 80 100 120 w1(c3) w2(c5) w3(c7) w4(c9) Number of Batches Processed over 10 Minutes Legend Wn = n number of workers or executors W2(c5) means 2 executors and a total of 5 cores across both executors