How to Build Real-Time Streaming Analytics with an In-memory, Scale-out SQL Database
Download as PPTX, PDF•2 likes•1,374 views
The document outlines the setup and requirements for building real-time streaming analytics with VoltDB, emphasizing the importance of fast data processing alongside traditional big data exploration. It discusses various features such as continuous query transactions, materialized views, and ranking indexes to facilitate efficient event processing and data aggregation. Additionally, it highlights integration capabilities with data sources and export targets to enhance analytics applications.
![page© 2015 VoltDB PROPRIETARY
EXTENSIBLE OPEN SOURCE API
22
public void onBlockStart() throws RestartBlockException;{
}
public boolean processRow(int rowSize, byte[] rowData) throws RestartBlockException {
}
public void onBlockCompletion() throws RestartBlockException {
}
VoltDB architecture
Commodity HW HA + ACID Scale-out VM-friendly](https://image.slidesharecdn.com/sqlwebinar042315final-150424131923-conversion-gate02/85/How-to-Build-Real-Time-Streaming-Analytics-with-an-In-memory-Scale-out-SQL-Database-22-320.jpg)
How to Build Real-Time Streaming Analytics with an In-memory, Scale-out SQL Database
- 1. page HOW TO BUILD REAL-TIME STREAMING ANALYTICS WITH AN IN-MEMORY, SCALE-OUT SQL DATABASE Ryan Betts, CTO VoltDB 1
- 2. page© 2015 VoltDB PROPRIETARY OUR SPEAKER Ryan Betts CTO at VoltDB 2
- 3. page © 2015 VoltDB PROPRIETARY page AGENDA • Setup: Fast vs. Big • Fast data application requirements • The role of analytics • Concrete examples 3
- 4. page© 2015 VoltDB PROPRIETARY Collect Explore AnalyzeAct 4 Big Data analytic results: 1. Discoveries: seasonal predictions, scientific results, long-term capacity planning 1. Optimizations: market segmentation, fraud heuristics, optimal customer journey
- 5. page© 2015 VoltDB PROPRIETARYEnterprise Apps ETL CRM ERP Etc. Data Lake (HDFS) BIG DATA Non Relational Processing BI Reporting Fast Operational Database FAST DATA Export Ingest / Interactive Real-time Analytics Fast Serve Analytics Decisioning Data Warehouse Columnar Analytics OLAP DATA ARCHITECTURE FOR FAST + BIG DATA
- 6. page© 2015 VoltDB PROPRIETARY 6 Fast (in motion) Streaming Analytics: real time summary and aggregation Transaction Processing: per-event decisions using context + history Big (at rest) Exploration: data science, investigation of large data sets Reporting: recommendation matrices, search indexes, trend and BI
- 7. page© 2015 VoltDB PROPRIETARY MODERN OLTP 7 1. Processing streams requires integrated access to state. 2. Using real time analytics requires a query interface. 3. Reacting to incoming events requires transactions. State + Query + Transactions = OLTP Fast Streaming Analytics Transaction Processing
- 8. page© 2015 VoltDB PROPRIETARY 8 Continuous Query Transactions Transformations • Materialized Views • Capped Tables • Ranking Indexes • Per-event Java + SQL • ACID processing • Millisecond latency responses • Loaders/Importers • Export Connectors • State for sessionization, enrichment VoltDB Architecture Commodity HW HA + ACID Scale-out VM-friendly
- 9. page© 2015 VoltDB PROPRIETARY MATERIALIZED VIEWS • Declarative SQL • Fully transactional • Supports ad-hoc query 9 CREATE VIEW registrations_by_zipcode ( zipcode, registered_voters ) AS SELECT zipcode, count(*) from voters where registration=1 GROUP BY zipcode;
- 10. page© 2015 VoltDB PROPRIETARY MV FOR STREAMING AGGREGATION • Partitioned on cluster • Immediately up-to-date • Active/active HA 10 Global Read: SELECT sum(count) WHERE sec > 130 and sec < 140;
- 11. page© 2015 VoltDB PROPRIETARY MATERIALIZED VIEWS WITH ACID TRANSACTIONS • Can be queried as part of a transaction • Example: fast quota enforcement 11 1-partition throughput (transactions/second) 10GB of data being aggregated.
- 12. page© 2015 VoltDB PROPRIETARY CAPPED COLLECTIONS • Simple windows • Durable, queryable • Support Mat. Views 12
- 13. page© 2015 VoltDB PROPRIETARY RANKING INDEXES FOR LEADERBOARDS • Sorted indexes are ordered statistic trees for O(log(n)) ranking • Quickly find overall rank • Quickly count items in range 13 SELECT COUNT(*) FROM scores WHERE score > 281; SELECT COUNT(*) FROM scores WHERE score >= 10 AND score <= 200;
- 14. page© 2015 VoltDB PROPRIETARY SQL SUPPORT 14 http://downloads.voltdb.com/documentation/TriFoldDevQuickRef.pdf • ALTER TABLE|CONSTRAINT|COLUMN|PROCEDURE • UNIQUE, MULTI-KEY INDEXES • INDEXES ON COLUMN FUNCTIONS • SQL ONLY DDL STORED PROCEDURES • JAVA STORED PROCEDURES • AUTO-GENERATED CRUD COMMANDS + REST API • MATERIALIZED VIEWS • SUBQUERY, UPSERT|INTO, JOIN, SELF-JOIN, INSERT SELECT • ~60 COLUMN FUNCTIONS
- 15. page© 2015 VoltDB PROPRIETARY COMBINED JAVA + SQL • Logic + SQL • 3rd party code 15 VoltDB architecture Commodity HW HA + ACID Scale-out VM-friendly
- 16. page© 2015 VoltDB PROPRIETARY ACID PROCESSING • Sync intra-cluster replication • Replicated durability • High availability (configurable) • Serializable isolation • Atomic ad-hoc or stored procedures • Partitioned & distributed txns • Load balanced reads across replicas 16
- 17. page© 2015 VoltDB PROPRIETARY ACID MATTERS • Speed of development • Richness of application • Obvious for billing, policy enforcement, authorization • Equally necessary for aggregation • Update in place desirable vs. batch process for ingest 17
- 18. page© 2015 VoltDB PROPRIETARY Performance – millisecond per-event responses SoftLayer: Update and Read Latency Latency(ms) Throughput (ops/sec) SoftLayer AWS YCSB Workload B – SoftLayer vs AWS
- 19. page© 2015 VoltDB PROPRIETARY INTEGRATING DATA SOURCES WITH VOLTDB • CSV loader • Kafka loader • JDBC loader • Vertica UDx • Extensible loader API • JDBC • ODBC • HTTP JSON • Native client drivers / SDKs BULK LOADERS APPLICATION INTERFACES
- 20. page© 2015 VoltDB PROPRIETARY VOLTDB EXPORT UI CREATE TABLE events ( EventID INTEGER, time TIMESTAMP, msg VARCHAR(128)); EXPORT TABLE events; 20 <export enabled="true" target="file"> ddl.sql deployment.xmlINSERT into TABLE values… Application SQL
- 21. page© 2015 VoltDB PROPRIETARY INTEGRATING VOLTDB WITH EXPORT TARGETS 21 • Local file system export • JDBC export • Kafka export • RabbitMQ export • HDFS export • HTTP export • Extensible API
- 22. page© 2015 VoltDB PROPRIETARY EXTENSIBLE OPEN SOURCE API 22 public void onBlockStart() throws RestartBlockException;{ } public boolean processRow(int rowSize, byte[] rowData) throws RestartBlockException { } public void onBlockCompletion() throws RestartBlockException { } VoltDB architecture Commodity HW HA + ACID Scale-out VM-friendly
- 23. page© 2015 VoltDB PROPRIETARY REVIEW Application Event Sources VoltDB Client Interface Partition Replica 1 Partition Replica 2 Export Destination (OLAP, HTTP) • SQL + Java transactions • JSON column values • HA in-memory processing • ACID (durable to disk) • Ranking indexes • Indexes on functions • Capped tables • Mat. views: RT aggregation • Append only export • 1-5 ms @ 99% responses
- 24. page© 2015 VoltDB PROPRIETARY BIGGER PICTURE 24
- 25. page© 2015 VoltDB PROPRIETARY 25
- 26. page© 2015 VoltDB PROPRIETARY QUESTIONS? • Use the chat window to type in your questions • Try VoltDB yourself: Download the Enterprise Edition: • www.voltdb.com/download Check out our Sample Apps: • www.voltdb.com/community/applications Open source version is available on github.com 26
- 27. page© 2015 VoltDB PROPRIETARY page THANK YOU! 27