KSQL---Streaming SQL for Apache Kafka
2 likes2,303 views
KSQL is a streaming SQL engine for Apache Kafka that allows users to perform stream processing by writing SQL-like queries. It enables non-engineers to process streaming data by using a familiar SQL syntax. KSQL queries can perform operations like filtering, aggregations, joins, and windowing on streaming data stored in Kafka topics. It leverages Kafka Streams to provide distributed, scalable, and fault-tolerant stream processing. KSQL can be run in client-server mode, embedded in applications, or deployed as standalone streaming ETL jobs.
KSQL---Streaming SQL for Apache Kafka
- 1. 1 KSQL – Streaming SQL for Apache Kafka Matthias J. Sax | Software Engineer [email protected] @MatthiasJSax
- 2. 2 1.0 Enterprise Ready A Brief History of Kafka and Confluent 0.11 Exactly-once semantics 0.10 Data processing (Streams API) 0.9 Data integration (Connect API) Intra-cluster replication 0.8 2012 2014 Cluster mirroring0.7 2015 2016 20172013 2018 CP 4.1 KSQL GA
- 3. 3 Why KSQL? • Enable Stream Processing for Non-Engineers • Everybody knows SQL • Look Ma, no code! • Declarative stream processing language • Fast prototyping • Streaming SQL engine for Apache Kafka • Streaming ETL • Ad-hoc topic inspection
- 4. 4 Trade-Offs • subscribe() • poll() • send() • flush() Consumer, Producer • filter() • join() • aggregate() Kafka Streams • Select…from… • Join…where… • Group by.. KSQL Flexibility Simplicity
- 6. 6 CREATE STREAM clickstream ( time BIGINT, url VARCHAR, status INTEGER, bytes INTEGER, user_id VARCHAR, agent VARCHAR) WITH ( value_format = 'JSON', kafka_topic = 'my_clickstream_topic' ); Creating a Stream
- 7. 7 Querying Streams CREATE STREAM user_clicks AS SELECT * FROM clickstream WHERE user_id = 'mjsax';
- 8. 8 CREATE TABLE clicks AS SELECT user_id, COUNT(url) FROM clickstream WINDOW TUMBLING (size 30 seconds) GROUP BY user_id HAVING COUNT(url) > 20 WHERE bytes > 1024; Windowed Aggregations
- 11. Do you think that’s a table you are querying ?
- 12. 12 CREATE TABLE users ( user_id INTEGER, registered_at LONG, username VARCHAR, name VARCHAR, city VARCHAR, level VARCHAR) WITH ( key = 'user_id', kafka_topic = 'clickstream_users', value_format = 'AVRO'); Creating a Table
- 13. 13 Confluent Schema Registry FTW! CREATE TABLE users WITH ( key = 'user_id', kafka_topic = 'clickstream_users', value_format = 'AVRO'); Creating a Table
- 14. 14 Using Tables
- 15. 15 CREATE STREAM vip_actions AS SELECT c.user_id, fullname, url, status FROM clickstream c LEFT JOIN users u ON c.user_id = u.user_id WHERE u.level = 'Platinum'; Joins for Enrichment
- 18. 18 How to deploy and use KSQL
- 19. 19 How to run KSQL JVM KSQL Server KSQL CLI JVM KSQL Server JVM KSQL Server Kafka Cluster #1 Client-server
- 20. 20 How to run KSQL #1 Client-server • Start any number of server nodes bin/ksql-server-start • Start one or more CLIs and point them to a server bin/ksql https://myksqlserver:8090 • All servers share the processing load Technically, instances of the same Kafka Streams Applications Scale up/down without restart
- 21. 21 How to run KSQL JVM KSQL Server JVM KSQL Server JVM KSQL Server #2 as a standalone Application Kafka Cluster
- 22. 22 How to run KSQL #2 as a standalone Application • Start any number of server nodes Pass a file of KSQL statement to execute bin/ksql-node query-file=foo/bar.sql • Ideal for streaming ETL application deployment Version-control your queries and transformations as code • All running engines share the processing load Technically, instances of the same Kafka Streams Applications Scale up/down without restart
- 23. 23 How to run KSQL #3 EMBEDDED IN AN APPLICATION JVM App Instance KSQL Engine Application Code JVM App Instance KSQL Engine Application Code JVM App Instance KSQL Engine Application Code Kafka Cluster
- 24. 24 How to run KSQL #3 EMBEDDED IN AN APPLICATION • Embed directly in your Java application • Generate and execute KSQL queries through the Java API Version-control your queries and transformations as code • All running application instances share the processing load Technically, instances of the same Kafka Streams Applications Scale up/down without restart
- 25. 25 Internals
- 26. 26 Internals Read input from Kafka Operator DAG: • filter/map/aggregation/joins • Operators can be stateful Write result back to Kafka
- 27. 27 Internals
- 30. 30 Scaling
- 31. 31 Take home • Streaming SQL engine for Apache Kafka • Leverages Kafka Streams • Open Source: Apache 2.0 • https://github.com/confluentinc/ksql/ • Included in Confluent Platform 4.1 • https://www.confluent.io/product/ksql/ • https://docs.confluent.io/current/ksql/docs/index.html
- 32. 32 Thank You We are hiring!