Sybase Complex Event Processing
4 likes4,433 views
Complex Event Processing (CEP) analyzes streams of event data to identify patterns and derive meaningful information. CEP allows for real-time situational awareness, immediate response, and better decision making using continuous intelligence. The Sybase Event Stream Processor is a CEP engine that can process unlimited input streams at high speeds with low latency. It is used for applications like fraud detection, system automation based on real-time conditions, streaming analytics, and continuous intelligence for better decisions.
Sybase Complex Event Processing
- 1. Complex Event Processing SAP ANALYTICS & INNOVATION FORUM '12 Bulut Altıntaş, Sybase, Teknik Danışman 16.03.2012
- 2. SYBASE: THE UNWIRED ENTERPRISE INFORMATION Creating Value at the INFORMATION MANAGEMENT Point of Decision MOBILITY Any Application Any Device DECISION-READY INFORMATION Any Server Platform Any Person INTEGRATION COLLABORATION Any Data Source Always Available MANAGE ANALYZE MOBILIZE © 2012 SAP AG. All rights reserved. 2
- 3. Complex Event Processing Overview What is Complex Event Processing? © 2012 SAP AG. All rights reserved. 3
- 4. Need For Complex Event Processing © 2012 SAP AG. All rights reserved. 4
- 5. INSIGHT FROM FAST-MOVING DATA CONTINUOUS INTELLIGENCE™ • Situational Awareness • Immediate Response • Better decision: complete, timely insight © 2012 SAP AG. All rights reserved. 5
- 6. A Simple CEP Example Music Detected + Event correlation, pattern Truck Approaching Derived result matching + = Children running across street Ice cream truck coming © 2012 SAP AG. All rights reserved. 6
- 7. CEP is used in four different ways – often in combination Real-time Business Intelligence System Automation Situation Detection Automated response Analyze live event streams for patterns Continuously adjust business which identify business critical situations processes to reflect current conditions Examples: Examples: • Fraud Detection • Automated pricing based on the • Customer Behavior Patterns market indicate a problem • Real-time Resource allocation Continuous Stream Intelligence Transformation Live dashboards that show actionable Cleanse and enrich live data streams, information for better decision making turning raw data into actionable info Examples: Examples: • Real-time P&L, Limits/Exposure Mgmt • Streaming alternative to ETL • Online marketing and offer performance • Filter and aggregate into summary • System monitoring information © 2012 SAP AG. All rights reserved. 7
- 8. The SYBASE Event Stream Processor (ESP) Unlimited number of input streams Send output to apps, dashboards Incoming data is processed as it arrives, Query on demand according to the business logic defined Interactive analysis using high level authoring tools © 2012 SAP AG. All rights reserved. 8
- 9. Key Components ESP Server – Manages the run-time components; starts and stops Projects – A “project” is like an application: it contains a set of stream, window, schema and query definitions – ESP server is designed for speed, scalability, and resilience ESP Studio – Eclipse® based environment used to build and test ESP Projects Adapters – Input adapters connect to an external data source and stream the data into an ESP stream or window – Output adapters connect to external consumers and stream ESP output to them Application Programming Interfaces (API) – Available for C/C++, Java and .NET Sybase RAP – Complementary product that integrates seamlessly with CEP engine for data capture, providing long term history that can be used for historical analysis, tick capture, ad-hoc analysis and compliance Panopticon EX Dashboard – Optional real-time dashboard providing visually rich, interactive data analysis across both streaming CEP data and historical data in RAP © 2012 SAP AG. All rights reserved. 9
- 10. Sybase ESP Architecture Command Interface, Applications, Stream Viewer, Third party clients, Resource Monitor Query Client - Platform management - Ad-hoc queries - Platform monitoring - Alerts via call-out functions C&C Connection Mgmt (ODBC, JDBC) (XML-RPC) SQL Security (PAM, SSL, Audits) Data Streams Stream Processor Data Streams • Multi-threaded I/O Gateway I/O Gateway • 64 bit Security Security Publish • High message rates Subscribe Adapters • High retention capacity Adapters Data Streams • Ultra-low latency Data Streams • Optional persistence Log Store Mem Store Config Authoring Tools © 2012 SAP AG. All rights reserved. 10 10
- 11. Sybase ESP Engine Performance STAC Certified measurement of throughput and latency Only CEP vendor to publish STAC benchmarks Linear scaling with addition of more cores: − 200,000 messages/sec on 16 cores − 300,000 messages/sec on 24 cores Consistent/predicable low latency. Only minor increase in latency from 20,000 to 300,000 messages/sec: − Mean: 1.296 - 1.568 ms − Medium: 1.177 - 1.502 ms − Std Dev: 0.567 - 1.104 ms Source: STAC Research Sep 08 − Min: 0.091 – 0.115 ms © 2012 SAP AG. All rights reserved. 11
- 12. ESP STUDIO: Both Visual & Texual Authoring Visual dataflow authoring based on Eclipse® © 2012 SAP AG. All rights reserved. 12
- 13. Key Concepts Input Streams Derived Streams Receive events from adapters Apply continuous query operators to one or more input streams to produce a new stream Database Analogy Streams can Have State • Streams are like tables • Retention windows define how many or • Events are rows in the table how long events are kept • Fields in an event message are • Opcodes in events can indicate columns in the table insert/update/delete and can be • Continuous Queries produce automatically applied to the window result sets/streams © 2012 SAP AG. All rights reserved. 13
- 14. CCL: CONTINUOUS COMPUTATION LANGUAGE SQL-BASED EVENT PROCESSING LANGUAGE • Leverage familiarity and simplicity of SQL • Instead of ‘snapshot’ queries, continuous queries Filter (WHERE), Aggregate (GROUP BY), Join, Compute • Extension for event stream Windows Patterns © 2012 SAP AG. All rights reserved. 14
- 15. Adapters - Rich Integration Options © 2012 SAP AG. All rights reserved. 15
- 16. Panopticon: Real Time Data Analysis © 2012 SAP AG. All rights reserved. 16
- 17. DATA MOVEMENT COMPONENTS Replicate Sybase ASE Sybase IQ Oracle Primary Catalog/Control DB Sybase ASE Oracle Replication RSSD MS SQL Server IBM DB2/OS390 IBM DB2/UDB Rep Agent Route Change Detection RepConnector SYBASE Replication ESP Server/ECO ECDA Gateway Rules Engine MS SQL Queues IBM DB2 Stable Devices Any ODBC Message Repository Sybase Control Center Monitoring © 2012 SAP AG. All rights reserved. 17
- 18. CEP Industry Mapping CEP Industry Mapping Capital Banking Retail Government Energy Telecom Markets Defense/Aerospace Algorithmic Trading Smart Order Routing Fraud Detection Energy Grid Management Real-time Profit and Loss Market Data Enrichment Retail – Online Commerce Intrusion Detection/Network Monitoring Real-time Risk Analysis Real-time Surveillance/ Customer Management Real-time Data Management Compliance © 2012 SAP AG. All rights reserved. 18
- 19. Using CEP and RAP together © 2012 SAP AG. All rights reserved. 19
- 20. Capital Markets Data Management WHERE WE PARTICIPATE… Sources Visualization & Reporting Market Data Connectivity • Competitive Time Series + Financial In-Memory Database Historical Repository Trading & Risk Apps Mathematical Tools trading strategy Event Processing Analytics Exchanges & IBM DB2 Consolidators • Real time risk management Microsoft Message Bus SQL Server Ticker Plants CEP RAP Oracle MATH TOOLS VISUALIZATION DATA & LIBRARIES TIBCO RV Panopticon Sybase ASE RogueWave IBM MQ Series MS Excel Microsoft MSMQ Reuters RMDS WOMBAT Bloomberg Sybase Products Partner Products © 2012 SAP AG. All rights reserved. 20
- 21. National Security Rules (Authoring) Input Streams Geographic Dashboards Data Vessel / Vehicle Location ? Reporting Tools Events Alerts Historical Profile © 2012 SAP AG. All rights reserved. 21
- 22. Telecom: BICS BICS is a joint-venture between Belgacom, Swisscom and MTN CHALLENGE The gain in storage space generated by Sybase • Act proactively on possible future events that would CEP’s data aggregation threaten their customers’ quality of service saves us from having to • Immediate visibility into service problems make substantial • Flexible real-time aggregation of data investments in additional servers, CPU, power, a SOLUTION more extensive database • Sybase CEP (Complex Event Processing) infrastructure, making RESULTS major investments that weren’t very future-proof • Quality guarantee thanks to real-time, proactive response would have meant going to emergency signals in periods of peak traffic through the same process • Real-time voice network monitoring and analytics for high again a few years later. QoS Laurent Missa, OSS Manager at BICS • Comprehensive network monitoring and visibility • Rapid, flexible analysis of real-time data on demand © 2012 SAP AG. All rights reserved. 22
- 23. Finance: BANKS.COM BANKS.com, Inc. - financial services portal CHALLENGE The Sybase Aleri CEP • Gain real-time insight on customer activity and traffic engine will allow us to monitor user behavior, trends threaten their customers’ quality of service traffic, and trends in real- • improve everyday business operations and marketing time, allowing us to campaigns provide a more targeted SOLUTION approach to our marketing and advertising • Sybase CEP (Complex Event Processing) programs in order to RESULTS provide a more unique • Reduces costs by enabling more effective and targeted experience. marketing and advertising programs Steve Ernst, Chief Technology • Allows for the management of marketing and advertising Officer programs in real-time • Enables immediate response to new trends and internet traffic patterns • Produces a better product for advertisers © 2012 SAP AG. All rights reserved. 23
- 24. Finance: TURKDEX Turkish Derivatives Exchange, Inc. CHALLENGE The combination of simplicity and power • Managing market and trading risk effectively provided by the • Establish a comprehensive trading system to detect abusive Continuous Computation trading patterns at high-speed market rates Language and the rapid • Speed the development and deployment of their integration capabilities of surveillance applications. Sybase CEP, gave our developers the critical SOLUTION tools to create the • Sybase CEP (Complex Event Processing) sophisticated applications RESULTS in extremely challenging schedules to meet • Creates a reliable, secure trading environment regulatory and corporate • Supports the increase in derivative market investments and needs. trading volume Abdullah Akoglu, • Delivers simplified surveillance mechanism to identify Lead Developer market abnormalities © 2012 SAP AG. All rights reserved. 24
- 25. Just A Few of Our CEP Customers © 2012 SAP AG. All rights reserved. 25
- 26. Thank you İletişim Bilgileri Bulut Altıntaş Teknik Danışman [email protected] +90 (0) 212 351 27 30