SingleStore & Kafka: Better Together to Power Modern Real-Time Data Architecture | Rohit Reddy, SingleStore
0 likes989 views
The document discusses the integration of Kafka and SingleStore to enhance modern real-time data architecture, emphasizing the growing importance of cloud services and real-time analytics. Key capabilities of SingleStore include ultra-fast data ingestion, real-time operational analytics, and user-defined transformations via pipelines. It also highlights use cases such as real-time fraud detection and IoT analytics, promoting SingleStore as a unified database for fast analytics across various data sources.
SingleStore & Kafka: Better Together to Power Modern Real-Time Data Architecture | Rohit Reddy, SingleStore
- 1. — Kafka & SingleStore: Better Together to Power Modern Real-Time Data Architecture Rohit Reddy Principal Solutions Engineer SingleStore
- 2. 2 Increasing Focus on Cloud and Real-Time Analytics 90% 75% 30% By 2022, public cloud services will be essential for 90% of data and analytics innovation By 2022, 75% of all workloads will move to Hybrid-Cloud By 2025, nearly 30% of all data generated will be real-time Gartner Top 10 Trends in Data and Analytics for 2020 McKinsey - Unlocking business acceleration in a hybrid cloud world, Aug 2019 IDC - Data Age 2025
- 3. 3 — Traditional Data Architecture System of Record System of Engagement Data Warehouse ODS ETL CDC Reporting Visualization Batch Engine Real-Time Engine Data Lake
- 4. 4 — Modern Real-Time Data Architecture System of Record System of Engagement Smart Apps System of Insight System of Intelligence CDC
- 5. 5 The Unified Database for Fast Analytics Data Warehouse Operational Database Transactional Workloads Fast Queries | Large Data Size Aggregation Fast Lookup | High Concurrency Simplifies the support of diverse workloads by reducing operational complexity Analytical Workloads
- 6. 6 Ultra-Fast Ingest Parallel, high-scale streaming data Ingest Super-Low Latency High Concurrency Blazing Fast Queries Unparalleled Scalability Billions of events/sec for immediate availability Sub-second latencies with immediate consistency Millions of real-time queries across tens of thousands of users Fast Analytics Fast analytics on dynamic data for complex analytical queries SingleStore - Key Capabilities for Fast Analytics
- 7. 7 — Kafka & SingleStore Meet Demands of Operational Analytics Real-Time Millions of records per second Consistent Exactly-once semantic Distributed, Fault Tolerant Parallel ingest Developer-Friendly Pub-Sub & SQL
- 8. 8 — Anatomy of SingleStore Pipelines Sequence SingleStore pulls for changes from a data source system. SingleStore pulls the data into its memory space (no commit) where a transform can be applied. The data is committed in a transaction (and in parallel) Pipelines Kafka SingleStore Data can be directly inserted into tables or pre-processed by a stored procedure Write to Kafka
- 9. 9 SingleStore Pipelines Creation 10M UPSERTS PER SECOND WITH KAFKA + SINGLESTORE CREATE OR REPLACE PIPELINE load_trade_data AS LOAD DATA KAFKA 'hostname:9092/trades' WITH TRANSFORM ('score_data.py','','') -- optional INTO TABLE live_predictions -- directly into tables INTO PROCEDURE trade_proc -- via a stored procedure FIELDS TERMINATED BY ','; START PIPELINE load_trade_data; ; —
- 10. 10 — SingleStore Kafka Pipelines SingleStore Cluster
- 11. 11 — SingleStore Transforms ● Build transforms using any language! ● Transforms are an optional user-defined program that receives data from a pipeline’s extractor and outputs modified data (JSON, Avro, CSV) ○ Examples: Data modification, aggregation, feature engineering, model execution, and more! ● Linux distribution must have the required dependencies to execute the transform ● Data streamed to the transform is byte-length encoded Stream Transform Load
- 12. 12 — SingleStore Stored Procedures DELIMITER // CREATE OR REPLACE PROCEDURE tweets_proc(batch QUERY(tweet JSON)) AS BEGIN INSERT IGNORE INTO tweets(tweet_id, tweet_user_id, tweet_text) SELECT tweet::tweet_id, tweet::tweet_user_id, tweet::tweet_text FROM batch; INSERT INTO retweets_counter(user_id, num_retweets) SELECT tweet::retweet_user_id, 1 FROM batch WHERE tweet::retweet_user_id IS NOT NULL ON DUPLICATE KEY UPDATE num_retweets = num_retweets + 1; END // DELIMITER ; ; ● Preprocess incoming data: cleansing, aggregation, filtering… ● Dispatch to multiple tables ● Cross-reference with dimension tables ● Integrity check ● Push to Kafka
- 13. 13 — SingleStore Push to Kafka ● Allows users to leverage SingleStore as a true Operational Data Hub with downstream decisioning ● “SELECT … INTO KAFKA …” runs a SELECT query, constructs Kafka message for each row in the result set, and publishes the messages to a Kafka topic ● Includes every column value in the result set’s row and separates the column values by a delimiter ● Configure security credentials within the statement easily SELECT col1, col2, col3 FROM t ORDER BY col1 INTO KAFKA 'host.example.com:9092/test-topic' FIELDS TERMINATED BY ',' ENCLOSED BY '"' ESCAPED BY "t" LINES TERMINATED BY '}' STARTING BY '{';
- 14. 14 — SingleStore Confluent Kafka Connector ● SingleStore Kafka Connect Connector on the Confluent Hub ● Integration with Confluent Kafka Connect to stream data into SingleStore ● Management and deployment capabilities of Confluent make this incredibly easy to get started ● Cloud-first: Kafka Connector sits Kafka-side, eliminating many potential security constraints 5X THAN JDBC CONNECTOR FASTER
- 15. Real-time fraud analytics for Credit card swipes in less than 50ms. Real-time geospatial insights with massive concurrency to manage 24/7 operations 300K Events per second Streaming analytics to drive proactive care and real-time recommendations IoT Analytics ingesting and analyzing data from over 1.2 Million smart meters 13x data growth moving from batch to near-real time visibility and analytics 3500+ Users 1.2M Smart meters analyzed 10M Upserts per second Tier-1 US Bank 50ms Real-time Fraud Detection Top Energy Company
- 16. SingleStore is the Unified Database for Fast Analytics on Any Data, Anywhere
- 17. 17 Learn Your Way — Get Started with $500 in Free Credits Today Go to singlestore.com/managed-service-trial ● Learn by Reading ○ docs.singlestore.com ● Learn by Engaging Peers ○ singlestore.com/forum ● Learn by Watching ○ youtube.com/singlestore ● Learn through Training ○ training.singlestore.com
- 18. — 18 Thank You Sales Please fill out the form if you need to learn more. For immediate sales help, call us at 1-855-463-6775 or email us at [email protected]. Enterprise Edition Support Are you encountering an issue and have an enterprise support? Submit a support request. U.S. Office Locations San Francisco (HQ) 534 Fourth Street San Francisco, CA 94107 Seattle 96 Union Street Seattle, WA 98101 Portland 700 SW Fifth Ave Portland, OR 97204