Confluent Platform 5.5 + Apache Kafka 2.5 => New Features (JSON Schema, Protobuf, Exactly-Once, REST Proxy, etc.)
0 likes365 views
Confluent Platform 5.5 introduces several new features to simplify event streaming development including adding Protobuf and JSON schema support throughout the platform, providing exactly-once semantics for non-Java clients, introducing administrative functions to the REST Proxy, expanding the functionality of ksqlDB with new aggregates and data types, and adding a ksqlDB flow view to Confluent Control Center for increased visibility of streaming applications. The release is also based on the latest Apache Kafka 2.5 version.
Confluent Platform 5.5 + Apache Kafka 2.5 => New Features (JSON Schema, Protobuf, Exactly-Once, REST Proxy, etc.)
- 1. What’s New in Confluent Platform 5.5 May 5th 2020 Kai Waehner Technology Evangelist [email protected] LinkedIn @KaiWaehner www.confluent.io www.kai-waehner.de
- 2. Poll 1 What Stack are you using Today in your Projects? ● Apache Kafka (Pub/Sub, Storage, Connect, Streams) ● Confluent Community (REST, Schema Registry, KSQL, non-Java Clients, …) ● Confluent Platform (Self-managed Deployment) ● Confluent Cloud (Fully-managed Service) ● I don’t use Kafka yet www.kai-waehner.de | @KaiWaehner
- 3. Ways to Deploy Confluent Platform The enterprise distribution of Apache Kafka VM Deploy on any platform on-prem or cloud Self Managed Software Fully Managed Software Confluent Cloud Apache Kafka re-engineered for the Cloud Available on the leading public clouds Also via MarketplaceAlso via Marketplace
- 4. Confluent Platform Dynamic Performance & Elasticity Auto Data Balancer | Tiered Storage Flexible DevOps Automation Operator | Ansible GUI-driven Mgmt & Monitoring Control Center Event Streaming Database ksqlDB Rich Pre-built Ecosystem Connectors | Hub | Schema Registry Multi-language Development Non-Java Clients | REST Proxy Global Resilience Multi-region Clusters | Replicator Data Compatibility Schema Registry | Schema Validation Enterprise-grade Security RBAC | Secrets | Audit Logs TCO / ROI Revenue / Cost / Risk Impact Complete Engagement Model Efficient Operations at Scale Unrestricted Developer Productivity Production-stage Prerequisites Partnership for Business Success Freedom of Choice Committer-driven Expertise Open Source | Community licensed Fully Managed Cloud ServiceSelf-managed Software Training Partners Enterprise Support Professional Services ARCHITECTOPERATORDEVELOPER BUYER Apache Kafka
- 5. Rapid Pace of Innovation to Enable Enterprises April 2020 CP 5.5 (based on AK 2.5) Data Compatibility ● Protobuf and JSON schema support throughout platform Multi-language Development ● Exactly-once semantics for non- Java clients ● Admin functions for REST Proxy (preview) Event Streaming Database ● ksqlDB 0.7 ● ksqlDB Flow View Security ● AD/LDAP authentication for clients July 2019 CP 5.3 (based on AK 2.3) Security ● Role-Based Access Control (preview) ● Secret Protection DevOps automation ● Kubernetes Operator ● Ansible Playbooks Management & Monitoring ● Control Center redesigned user interface ● New CLI April 2019 CP 5.2 (based on AK 2.2) Developers ● Free single-broker developer license ● librdkafka and clients 1.0 KSQL ● New query expressions ● GUI enhancements Replicator ● Schema migration to CCloud Control Center ● Dynamic broker configuration ● Schema Registry management ● Multi-cluster Connect & KSQL ● Enhanced scalability January 2020 CP 5.4 (based on AK 2.4) Security ● Role-Based Access Control ● Structured Audit Logs Resilience ● Multi-Region Clusters Data Compatibility ● Schema Validation Management & Monitoring ● Control Center ○ RBAC management ○ Replicator monitoring Performance & Elasticity ● Tiered Storage (preview) Stream Processing ● ksqlDB features (preview) 5
- 6. What’s New in Confluent Platform 5.5
- 7. CP 5.5 makes Apache Kafka and event streaming more broadly accessible to developers Rich Pre-built Ecosystem Protobuf, JSON, and plug-in schema support for Schema Registry and throughout the platform 7 Multi-language Development Exactly-once semantics for non-Java clients Admin functions for REST Proxy (preview) Event Streaming Database ksqlDB 0.7 ksqlDB Flow View within Control Center
- 8. JSON Schema and Protobuf
- 9. Allow developers and businesses to adopt event streaming broadly Organizations no longer need to adjust event streaming apps to use Avro to use Schema Registry Applications with other formats also need to work with Schema Registry Plug-ins provide a way for Kafka users to address the “long tail” of data formats Protobuf and JSON are both very prevalent data formats The two data formats were the most commonly requested by customers by a significant margin Why add Protobuf, JSON Schema, and plug-in schema support to Schema Registry? 9 Rich Pre-built Ecosystem
- 10. 10 Schema Registry expands the Kafka ecosystem with Protobuf, JSON, and plug-in schema support Schema Registry Custom Plug- Ins Supported Data Formats Rich Pre-built Ecosystem App 1 ! Schema Registry Kafka topic ! Serializer App 1 Serializer
- 11. Kafka Connect Converters for Protobuf and JSON ksqlDB and Kafka Streams Enrich messages in either data format REST Proxy Register and retrieve schemas Control Center Create topic schemas directly in the GUI Schema Validation Broker-side enforcement of schemas 11 Protobuf and JSON Schema are also supported across all components of Confluent Platform Rich Pre-built Ecosystem
- 13. Why add exactly-once semantics to non-Java clients? 13 Exactly-once semantics ensures critical messages are neither skipped nor repeated For example, a message that could trigger a financial trade cannot be produced or consumed more than once Feature parity between the clients make event streaming accessible to all developers Organizations with many developer teams need a broad range of clients to allow Kafka to become their central nervous system Multi-language Development
- 14. Confluent enhances multi-language development with exactly-once semantics for librdkafka-based clients 14 Clients with EOS 0 1 2 3 4 5 6 7 8 Producer Write idempotently Read exactly once Consumer Multi-language Development
- 15. REST Proxy v3
- 16. Why introduce Admin functions for REST Proxy? 16 Removes need to switch over from REST Proxy to Java Admin client, CLI, or Control Center to run Admin functions REST Proxy can now both perform administrative tasks and produce and consume messages Feature parity between REST Proxy and the clients provides greater flexibility and more choices to developers Many developers prefer REST because of its language-agnostic nature and simplicity Multi-language Development
- 17. REST Proxy introduces Admin functions to improve Kafka connectivity (Preview State) 17 REST Proxy REST Proxy Applications Schema Registry REST / HTTP Describe and list brokers Create, delete, describe, and list topics Describe and list configs for topics New Admin Functions Multi-language Development
- 18. ksqlDB
- 19. Kafka producer/ consumer Kafka Streams ksqlDB The 3 stream processing modalities with Confluent
- 20. ConsumerRecords<String, String> records = consumer.poll(100); Map<String, Integer> counts = new DefaultMap<String, Integer>(); for (ConsumerRecord<String, Integer> record : records) { String key = record.key(); int c = counts.get(key) c += record.value() counts.put(key, c) } for (Map.Entry<String, Integer> entry : counts.entrySet()) { int stateCount; int attempts; while (attempts++ < MAX_RETRIES) { try { stateCount = stateStore.getValue(entry.getKey()) stateStore.setValue(entry.getKey(), entry.getValue() + stateCount) break; } catch (StateStoreException e) { RetryUtils.backoff(attempts); } } } The 3 stream processing modalities differ in flexibility and ease of use Kafka producer/consumer Kafka Streams ksqlDB builder .stream("input-stream", Consumed.with(Serdes.String(), Serdes.String())) .groupBy((key, value) -> value) .count() .toStream() .to("counts", Produced.with(Serdes.String(), Serdes.Long())); SELECT x, count(*) FROM stream GROUP BY x EMIT CHANGES;
- 21. Using external processing systems leads to complicated architectures DB CONNECTOR CONNECTOR APP APP DB STREAM PROCESSING CONNECTOR APPDB
- 22. We can put it back together in a simpler way DB APP APP DB APP PULL PUSH CONNECTORS STREAM PROCESSING STATE STORES ksqlDB
- 23. Connect integration and pull queries enable end-to-end streaming in just a few SQL statements Serve lookups against materialized views Create materialized views Perform continuous transformations Capture data CREATE STREAM purchases AS SELECT viewtime, userid,pageid, TIMESTAMPTOSTRING(viewtime, 'yyyy-MM-dd HH:mm:ss.SSS') FROM pageviews; CREATE TABLE orders_by_country AS SELECT country, COUNT(*) AS order_count, SUM(order_total) AS order_total FROM purchases WINDOW TUMBLING (SIZE 5 MINUTES) LEFT JOIN user_profiles ON purchases.customer_id = user_profiles.customer_id GROUP BY country EMIT CHANGES; SELECT * FROM orders_by_country WHERE country='usa'; CREATE SOURCE CONNECTOR jdbcConnector WITH ( ‘connector.class’ = '...JdbcSourceConnector', ‘connection.url’ = '...', …);
- 24. Why expand the functionality of ksqlDB? 24 ksqlDB is becoming the tool of choice to build event streaming applications that leverage stream processing ksqlDB continues to be adopted widely to create these modern applications Expanding the feature set further simplifies the process of building event streaming applications Adding more aggregates, support for more data types, etc. Event Streaming Database
- 25. Confluent Platform 5.5 launches with ksqlDB 0.7 to further simplify the development of event streaming applications 25 COUNT_DISTINCT Aggregate Count distinct values within a field COUNT_DISTINCT → 3 Support for more data types Record keys can now be of type String, INT, BIGINT, or DOUBLE Higher availability of pull queries1 Reroute queries to replica table partitions if the leader partition is down Pull Query 1 - Pull queries are still in preview Event Streaming Database
- 26. Why add the ksqlDB Flow View to Confluent Control Center? 26 Provides developers with a simple mental model of how tables, streams, and queries are interacting Applications often have many different queries that are more easily understood visually than reviewing the editor Enables developers to quickly discover and understand a ksqlDB application that others have built ksqlDB applications are often built by many different developers and teams, so discoverability is crucial Event Streaming Database
- 27. Control Center provides greater visibility and discovery of ksqlDB applications 27 Visualize what queries, streams, and tables are being used in a ksqlDB application ksqlDB Flow View Application Overview Event Streaming Database
- 28. Control Center provides greater visibility and discovery of ksqlDB applications 28 Drill down into table or stream nodes to quickly visualize their schema and messages ksqlDB Flow View Table / Stream Node Doubleclick Event Streaming Database
- 29. Drill down into query nodes to view the code and understand how inputs are being manipulated Control Center provides greater visibility and discovery of ksqlDB applications 29 ksqlDB Flow View Query Node Doubleclick Event Streaming Database
- 30. Confluent Platform 5.5 launches with the latest Apache Kafka 2.5 version
- 32. Release Notes ● https://docs.confluent.io/current/release-notes/index.html ● https://downloads.apache.org/kafka/2.5.0/RELEASE_NOTES.html … for all the details.
- 33. Poll 2 What Features from Confluent Platform 5.5 are the Most Exciting for You? ● Exactly-once Semantics for all Client APIs ● ksqlDB (including Connect Integration + Pull Queries) ● Improved UIs in Control Center ● New REST Proxy v3 ● Support for JSON Schema / Protobuf for whole Confluent Platform www.kai-waehner.de | @KaiWaehner