Complex Event Processing with Esper
29 likes•4,399 views
The document explains complex event processing (CEP), emphasizing its ability to analyze and respond to real-time data streams, often measured in nanoseconds. It highlights various application use cases across industries such as retail and finance, and differentiates between terminology like event stream processing (ESP) and simple event processing (SEP). Additionally, it outlines CEP architecture, event types, processing languages, and key considerations for developing high-performance CEP applications.
![EPL Statement Sample
select Part.zone from pattern [
every Part=CountZone(cnt in (1, 2)) ->
( timer:interval(1 min) and not
CountZone(zone=Part.zone, cnt in (0, 3))
)
]](https://image.slidesharecdn.com/complexeventprocessingwithesper-120117064536-phpapp01/85/Complex-Event-Processing-with-Esper-77-320.jpg)
![Listener and Engine
// Get engine instance and register statement
EPServiceProvider engine = EPServiceProviderManager.getDefaultProvider();
EPStatement statement = engine.getEPAdministrator().createEQL("...");
// Attach a listener
statement.addListener(new UpdateListener() {
public void update(EventBean[] newEvents, EventBean[] oldEvents) {
// Handle complex event
...
}
});](https://image.slidesharecdn.com/complexeventprocessingwithesper-120117064536-phpapp01/85/Complex-Event-Processing-with-Esper-78-320.jpg)
![Event filtering
// Filter for location report by location rectangle
select * from LR(x in [4:10], y in [6:12])](https://image.slidesharecdn.com/complexeventprocessingwithesper-120117064536-phpapp01/85/Complex-Event-Processing-with-Esper-80-320.jpg)
![Historical or reference data
// Alert when we hit the minimum inventory
// for a given zone
select zone, count(*)
from LR.std:unique(assetId) as lr,
sql:db[select mini from Minimum where zone=${lr.zone}]
having mini < count(*)](https://image.slidesharecdn.com/complexeventprocessingwithesper-120117064536-phpapp01/85/Complex-Event-Processing-with-Esper-84-320.jpg)
Complex Event Processing with Esper
- 1. Complex Event Processing with Esper Real-time Event Stream & Complex Event Processing Ted Won 2011. 11. 12. http://tedwon.com
- 5. Nanosecond ● CEP 세계의 측정 단위는 nanosecond ● A nanosecond (ns) is one billionth of a second
- 6. Nanosecond 1 second is... 1,000 ms 1,000,000 us 1,000,000,000 ns
- 7. Concept of CEP ● 교회에서... ● 벨이 울린다. ● 턱시도를 입은 남자와 꽃을 든 여자가 함께 걷는다. ● 두 사람 위로 꽃 잎이 날리고 푹죽이 터진다.
- 8. Concept of CEP CEP 기술은 이러한 Complex한 Event를 참조하여 결혼식이라는 것을 알아 차린다.
- 9. Imagination Beyond Reality ● Imaginary CEP Use Cases
- 10. Imagination Beyond Reality ● E-mart에서... ● 고객들이 카트를 끌고 매장을 돌아 다닌다. ● 카트의 이동 경로와 패턴을 알 수 없을까? ● 계산대에서 대기하는 고객들 상태 파악 후 실시간 조치 ● 카트 + 멤버십 카드라면 카트에 달린 모니터를 통해 개인화된 실시간 추천/쿠폰 발행 ● 카트에 물건을 담으면 자동으로 계산
- 11. Imagination Beyond Reality ● 전국의 이마트 매장이 130개... ● 엄청난 양의 데이터를 어찌 실시간으로 처리한단 말인가?? ● Starbucks 매장에서... ● 파주 프리미엄 아웃렛 매장에서... ● 교통 관리에서 실시간으로 교통 상태/패턴을 인지하고 적절한 조치
- 12. Imagination Beyond Reality ● Algorithmic Stock-Trading ● Applying CEP to the RHQ alert feature as a Operational Intelligence (OI) solution
- 13. What is CEP?
- 14. What is CEP ● Complex Event Processing ● CEP는 Event 중심의 실시간 대용량 데이터 처리 기술 ● CEP는 실시간 대용량 Event 처리 기술 ● CEP는 EDA(Event-driven architecture) 기반 시스템 ● Mutliple event stream에서 발생하는 패턴 감지 기술
- 15. What is CEP ● Identifying the most meaningful events from event cloud ● Real-time computation and pattern detection over event streams ● Response가 아닌 Reaction을 하는 시스템 ● http://en.wikipedia.org/wiki/Complex_Event_Processing
- 16. What is CEP ● Real-time Processing ● Intelligence System ● Real-time Business Intelligence ● Event Stream Intelligence
- 17. What is CEP ● Extreme Transaction Processing ● Publish/Subscribe Model ● Loosley-coupled ● Asynchronous Style Source: Event Processing in Action
- 18. Open Source CEP ● EsperTech Esper (Stream-oriented) ● JBoss Drools Fusion (Rule-oriented)
- 19. CEP Language Style ● Strem-oriented ○ EsperTech Esper ○ MS StreamInsight ● Rule-oriented ○ JBoss Drools Fusion ○ TIBCO TIBCO BusinessEvents
- 20. What is EDA?
- 21. What is EDA ● Event Driven Architecture ● EDA는 event의 production, detection, consumption과 reaction 하기위한 구조를 정의한 software architecture Source: Event Processing in Action
- 22. What is EDA ● More Loosely-coupled than SOA ● Event 표준 스펙 논의가 OASIS에서 Common Base Events라는 이름으로 진행 중 ● Response가 아닌 Reaction을 하는 시스템 ● 선처리 후저장 패러다임 ● http://en.wikipedia.org/wiki/Event-driven_architecture
- 23. Classical Request-response Architecture ● Synchronous interactions ● Request에 대한 Response를 하는 구조 ● 선저장 후처리 패러다임
- 24. Event Processing Styles ● SEP - Simple Event Processing ○ Single event based ○ 상태가 없는 Event를 처리하는 케이스 ○ JMS(Java Message Service)가 대표적인 예 ○ ESB(Enterprise Service Bus) ● ESP - Event Stream Processing ● CEP - Complex Event Processing
- 25. Event Processing Application Source: Event Processing in Action
- 26. ESP vs. CEP ● ESP - Event Stream Processing ○ Monitor streams of event data, analyse those events, and act upon opportunities ○ Volume weighted average of Google stock over the last (moving) 30min ● CEP - Complex Event Processing ○ Detecting patterns among events ○ If this Google VWAP increased more than 5% two times followed by Yahoo! VWAP decreased more than 10% then…
- 27. SOA vs. EDA ● SOA ○ RPC calls ○ Point-to-Point communication ○ Request and reply ○ Vertical interaction ● EDA ○ Message based ○ Less coupled than SOA ○ Publish and subscribe ○ Horizontal communication between tiers in process chain
- 28. Point-To-Point
- 30. What is an Event?
- 31. What is an Event ● Event란 무엇인가? ● 무엇을 Event로 정의 할 수 있는가?
- 32. What is an Event ● Something that happens ● 실제로 발생한 사건, 일, 메세지 ● 상태의 변경
- 33. What is an Event ● An event is an immutable record of a past occurrence of an action or state change. ● Event properties capture the useful information for an event.
- 34. What is an Event Event란 어떤 것이 변화한 것이다!
- 35. What is an Event ● 변화한다는 것은 어떠한 상태를 가지고 있다는 의미 ● 상태를 가지고 변화하는 모든 것이 Event로 정의될 수 있다. ● 이 세상의 모든 것은 변화한다. ● 변화량을 정량적으로 측정가능하도록 정의해야함 ● Event를 추상화하여 표현한 것을 Event type이라함
- 36. What is an Event ● 상태가 없는 Event를 처리하는 경우도 존재 ● SEP(Simple Event Processing)의 경우 존재 ● JMS가 하나의 예
- 37. Event Examples ● Stock tick ● 비밀번호 변경 ● 10초마다 실내 온도 측정 ● 서비스 응답 시간 ● 고객이 들고 다가오는 Starbucks membership cards with RFID ● 고객이 끌고 다니는 E-mart carts with RFID
- 38. Events vs. Messages ● Event는 때로는 Message로 표현됨 ● Message는 단순히 Event의 다른 표현이라는 측면도 존재 ● Event의 상태라는 속성이 Message에서는 어떤 내용(contents)이 라는 차이 ● EDA에 속한 개념으로써 SEP(Simple Event Processing)의 한 종류 인 JMS와 같은 기술에서 사용하는 전문 용어라는 측면
- 39. What is Event Type?
- 40. What is Event Type ● Event Representation Schema ● 개념적인 Event를 실제 프로그래밍적으로 표현하는 방법 ● Event의 개념적 표현을 Event Schema라 할 수 있는데 ● Event Schema의 표현을 Event Type 으로 한다. ● DBMS의 테이블 스키마와 유사한 개념 ● 대표적인 Data Format을 이용하여 정의 할 수 있다.
- 41. Event Type Data Format ● POJO ● Map ● XML ● JSON ● CSV
- 42. What is Event Stream?
- 43. What is Event Stream ● 시간의 순서대로 연속되는 Event ● 시작과 끝이 없는 Event의 연속된 흐름
- 45. What is Real-time Real-time system은 다음의 조건들을 충족해야 합니다 ● Low latency ● Regularities in Response time ● Providing predictable performance
- 46. CEP Usages ● Algorithmic Stock-Trading ● Operational Intelligence(OI) solutions ● Real-time analytics ● Predictive analytics ● Event-driven business process management ● BAM - Business Activity Monitoring ● BI - Business intelligence ● BRMSs - Business rule management systems ● Network and System Management ● MOM- Message-oriented middleware ● Stream computing ● ETL(Extract, Transform, Load)
- 47. CEP Products
- 48. Esper
- 49. EsperTech Esper ● Event Stream and Complex Event Processing platform ● Convergence of ESP and CEP ● Open Source ESP/CEP Engine ● Event Processing Language (EPL) ● Java and .Net
- 50. EsperTech Esper ● High Throughput, Low Latency ● Highly Available (EsperHA) ● Rich Multi-Window GUI (Enterprise Edition) ● Real-time Displays (Enterprise Edition) ● Enterprise *ilities (Enterprise Edition)
- 51. EsperTech Esper ● Usability ● Maintainability ● Scalability ● Availability and Reliability ● Extensibility ● Security ● Portability
- 52. EsperTech Esper ● Technical Support (Enterprise Edition) ● Lightweight & Embeddable ● No Runtime Support ● GNU GPL(General Public License) v2
- 53. Runtime: Esper + Java EE platform
- 54. Runtime: Esper + OSGi
- 55. Esper Documentation ● Esper Documentation Home ○ http://goo.gl/isNb6 ● Esper 4.4.0 Reference documentation ○ http://goo.gl/5TXel ● EsperIO 4.4.0 Reference documentation ○ http://goo.gl/76uOe
- 57. Esper EDAS
- 58. CEP Architecture
- 59. CEP Architecture
- 60. CEP Architecture
- 62. Real-time Log Aggregator & CEP
- 63. Input / Output Adapters ● Binary File Reader Input ● Binary File Writer Output ● CSV File Reader Input ● CSV File Writer Output ● CSV Socket Reader Input ● CSV Socket Writer Output ● Email (IMAP) Reader Input ● Email Sender ● HA Heartbeat Input ● IP Packet Capture /HTTP ● IP Packet Capture /IRC
- 64. Input / Output Adapters ● IP Packet Capture /POP3 ● IP Packet Capture /SMTP ● IRC Output adapter ● Jabber Output Adapter ● JDBC ● JMS Publish/Subscribe* ● Log Output ● Microsoft Excel RTD* ● Multicast UDP Input and Output* ● Quartz Enterprise Scheduler Input*
- 65. Input / Output Adapters ● Regular Expression File Reader Input ● Regular Expression Socket Reader Input ● RSS ● Twitter ● Unicast UDP Input Adapter ● XML File Writer Output ● XML over HTTP Writer Output ● Hadoop HDFS Input Adapter ● Hadoop HDFS Output Adapter
- 66. Esper EPL
- 67. Event Processing Language select symbol, symbolDesc from OrderEvent.win:time(30 min)
- 68. Event Processing Language ● No standard ● SQL-like language ● Data windows ● Pattern matching ● Continuous query
- 74. Event Processing Language ● Event filtering ● Sliding windows and aggregation ● Grouped windows and output rate limiting ● Joins and Outer Joins ● Historical or reference data ● Subquery
- 75. Event Example: The RFID Domain ● RFID: Radio Frequency Identification ● LocationReport ○ asset id - a unique identifier of the tagged asset ○ x - the x location value ○ y - the y location value ○ zone - derived from x and y ● Use cases ○ “When a given group of assets are not moving together from zone to zone, then...” ○ “When a given asset stays too long in the same zone, then...”
- 76. EPL Statement Sample ● If a given set of assets are not moving together from zone to zone, ● then alert insert into CountZone select zone, count(*) as cnt from LocationReport.std:unique('assetId') where assetId in (1, 2, 3) group by zone
- 77. EPL Statement Sample select Part.zone from pattern [ every Part=CountZone(cnt in (1, 2)) -> ( timer:interval(1 min) and not CountZone(zone=Part.zone, cnt in (0, 3)) ) ]
- 78. Listener and Engine // Get engine instance and register statement EPServiceProvider engine = EPServiceProviderManager.getDefaultProvider(); EPStatement statement = engine.getEPAdministrator().createEQL("..."); // Attach a listener statement.addListener(new UpdateListener() { public void update(EventBean[] newEvents, EventBean[] oldEvents) { // Handle complex event ... } });
- 79. Sending events // Get the same engine instance EPServiceProvider engine = EPServiceProviderManager. getDefaultProvider(); EPRuntime runtimeEngine = engine.getRuntime(); ... LocationReport event = new LocationReport(assetId,x,y,zone); runtimeEngine.sendEvent(event);
- 80. Event filtering // Filter for location report by location rectangle select * from LR(x in [4:10], y in [6:12])
- 81. Sliding windows and aggregation // Count all assets reporting zone 10 in last 30 sec select count(*) from LR(zone=10).win:time(30 sec)
- 82. Grouped windows and output rate limiting // Get X location range of the last 100 events per zone // every 1 minute select zone, min(x), max(x) from LR.std:groupby('zone').win:length(100 events) output every 1 min
- 83. Continuous joins // Fire when any asset enters zone 2 before zone 1 select Zone2.assetId from LR(zone=2).win:time(1 day) Zone2 left outer join LR(zone=1).win:time(1 day) Zone1 on Zone1.assetId = Zone2.assetId where Zone1.assetId is null
- 84. Historical or reference data // Alert when we hit the minimum inventory // for a given zone select zone, count(*) from LR.std:unique(assetId) as lr, sql:db[select mini from Minimum where zone=${lr.zone}] having mini < count(*)
- 85. Subquery // Notify when the asset id is not known select * from LR where assetId not in (select assetId from KnownAsset.std:unique('assetId'))
- 86. Data window views ● win:length ● win:length_batch ● win:time ● win:time_batch ● win:time_length_batch ● win:time_accum ● win:ext_timed ● ext:sort_window ● ext:time_order
- 87. Data window views ● std:unique ● std:groupwin ● std:lastevent ● std:firstevent ● std:firstunique ● win:firstlength ● win:firsttime
- 88. Statistics views ● std:size ● stat:uni ● stat:linest ● stat:correl ● stat:weighted_avg
- 89. CEP Application Development Steps ● Define Event ● Define Event Type ● Define EPL Statements ● Publish EPL Statements ● Subscribe Output Adapter ● Attache Input Adapter
- 90. CEP Application Development Points ● Runtime 환경 결정 ● Resilience를 지원 여부 ● Zero-downtime event processing ● Support dynamic EPL Statement updates ● Input/Output Adapter에서 bottleneck이 발생 ● 그러므로 Adapter의 성능이 중요 ● Event의 타입과 용량에 따라서 성능에 영향 ● Input Adapter의 통신 protocol 결정 중요 ● CEP Engine Performance Tuning
- 91. CEP Application Key Considerations ● Low Latency ● High Throughput ● High Availability - Failover ● Zero-downtime
- 92. Performance ● Throughput ○ events/sec ● Latency ○ us/event ● 성능 측정은 throughput을 고정하고 latency statistics로 측정 ● http://docs.codehaus.org/display/ESPER/Esper+performance
- 93. Latency statistics ● Throughput 995900 (active 0 pending 0) ---Stats - engine (unit: ns) Avg: 2656 #9936457 0< 5000: 99.78% 99.78% #9914178 5000 < 10000: 0.20% 99.98% #20120 10000 < 15000: 0.01% 99.98% #646 15000 < 20000: 0.00% 99.99% #69 20000 < 25000: 0.00% 99.99% #36 25000 < 50000: 0.00% 99.99% #56 50000 < 100000: 0.00% 99.99% #26 100000 < 500000: 0.00% 99.99% #189 500000 < 1000000: 0.00% 99.99% #2 1000000 < 2500000: 0.00% 99.99% #68
- 94. Thanks Q&A
- 95. Thanks Q&A
- 97. JBoss Community (http://www.jboss.org) Korea JBoss User Group (http://cafe.naver.com/jbossug)