End to-end large messages processing with Kafka Streams & Kafka Connect
2 likes757 views
This document discusses processing large messages with Kafka Streams and Kafka Connect. It describes how large messages can exceed Kafka's maximum message size limit. It proposes using an S3-backed serializer to store large messages in S3 and send pointers to Kafka instead. This allows processing logic to remain unchanged while handling large messages. The serializer transparently retrieves messages from S3 during deserialization.
End to-end large messages processing with Kafka Streams & Kafka Connect
- 1. End-to-end large messages processing with Kafka Streams & Kafka Connect | Philipp Schirmer | bakdata GmbH | March 19th, 2020 End-to-end large messages processing with Kafka Streams & Kafka Connect Philipp Schirmer | March 19th, 2020
- 2. End-to-end large messages processing with Kafka Streams & Kafka Connect | Philipp Schirmer | bakdata GmbH | March 19th, 2020 Text-heavy data Icons made by Pause08, Freepik, Zlatko Najdenovski, monkik from www.flaticon.com
- 3. End-to-end large messages processing with Kafka Streams & Kafka Connect | Philipp Schirmer | bakdata GmbH | March 19th, 2020 Load data into Kafka, process and index in Elasticsearch Amazon S3 Elasticsearch https://aws.amazon.com/architecture/icons/ https://www.seekpng.com/ipng/u2q8r5r5w7e6q8t4_white-on-transparent-kafka-logo-svg/ https://cdn.worldvectorlogo.com/logos/elasticsearch.svg
- 4. End-to-end large messages processing with Kafka Streams & Kafka Connect | Philipp Schirmer | bakdata GmbH | March 19th, 2020 Large messages in Kafka NLP-Operations require full document But: ● Kafka prefers many small messages ● max.message.bytes defined by brokers (default 1MB) ● Limit cannot be configured in some scenarios (e.g., Confluent Cloud) ● There will always be (few) messages exceeding the limit
- 5. End-to-end large messages processing with Kafka Streams & Kafka Connect | Philipp Schirmer | bakdata GmbH | March 19th, 2020 Solutions: Chunking Split document into multiple small parts
- 6. End-to-end large messages processing with Kafka Streams & Kafka Connect | Philipp Schirmer | bakdata GmbH | March 19th, 2020 Solutions: Chunking Split document into multiple small parts But: requires change of processing logic ● User must be aware of partial documents ● Expensive aggregations required
- 7. End-to-end large messages processing with Kafka Streams & Kafka Connect | Philipp Schirmer | bakdata GmbH | March 19th, 2020 Solutions: External Storage Store large messages in external system, e.g., Amazon S3, and send pointers to Kafka s3://
- 8. End-to-end large messages processing with Kafka Streams & Kafka Connect | Philipp Schirmer | bakdata GmbH | March 19th, 2020 Kafka Streams SerDe Kafka stores binary messages SerDes offer an API to interpret raw messages with Kafka Streams (Java) KStream<String, String> stream = builder.stream("my_topic", Consumed.with(Serdes.String(), Serdes.String()));
- 9. End-to-end large messages processing with Kafka Streams & Kafka Connect | Philipp Schirmer | bakdata GmbH | March 19th, 2020 S3-backed SerDe Our S3-backed SerDe transparently handles storage and retrieval of messages ● Delegates actual (de-)serialization to a wrapped SerDe ● No changes to processing logic required, only configuration changes ● Small messages are sent to Kafka (almost) as before Similar implementations for Kafka Connect and Faust (Python) bakdata/kafka-s3-backed-serde (GitHub) bakdata/faust-s3-backed-serializer (GitHub)
- 10. End-to-end large messages processing with Kafka Streams & Kafka Connect | Philipp Schirmer | bakdata GmbH | March 19th, 2020 Serialization 1. Serialize message using actual SerDe 2. Check if message size in bytes exceeds configured limit 3. a. If message is small enough i. Send actual message small message flag to Kafka b. If message is too large i. Store message on S3 ii. Send S3 URI and large message flag to Kafka
- 11. End-to-end large messages processing with Kafka Streams & Kafka Connect | Philipp Schirmer | bakdata GmbH | March 19th, 2020 Serialization - S3 URI S3 URIs must be unique s3://my-bucket/large-messages/my-topic/values/12ef79ab1097ebca1abe5982ce1763aa s3backed.base.path key/value topic unique id s3backed.id.generator ● Random UUID ● Murmur hash ● SHA-256
- 12. End-to-end large messages processing with Kafka Streams & Kafka Connect | Philipp Schirmer | bakdata GmbH | March 19th, 2020 Deserialization 1. Check flag 2. a. If flag denotes small message i. Proceed b. If denotes large message i. Retrieve message from S3 3. Deserialize message using actual SerDe
- 13. End-to-end large messages processing with Kafka Streams & Kafka Connect | Philipp Schirmer | bakdata GmbH | March 19th, 2020 Demo https://github.com/bakdata/s3-backed-serde-demo Amazon S3 Elasticsearch https://aws.amazon.com/architecture/icons/ https://www.seekpng.com/ipng/u2q8r5r5w7e6q8t4_white-on-transparent-kafka-logo-svg/ https://cdn.worldvectorlogo.com/logos/elasticsearch.svg
- 14. End-to-end large messages processing with Kafka Streams & Kafka Connect | Philipp Schirmer | bakdata GmbH | March 19th, 2020 tf-idf … is a numerical statistic that is intended to reflect how important a word is to a document in a collection or corpus. https://en.wikipedia.org/wiki/Tf%E2%80%93idf https://de.wikipedia.org/wiki/Tf-idf-Ma%C3%9F
- 15. End-to-end large messages processing with Kafka Streams & Kafka Connect | Philipp Schirmer | bakdata GmbH | March 19th, 2020 S3-backed SerDe - Retention Kafka topics have a message retention S3 supports object expiration ● Each rule applies to a prefix → each topic has its own prefix ● Timestamp of S3 object and Kafka message can differ
- 16. End-to-end large messages processing with Kafka Streams & Kafka Connect | Philipp Schirmer | bakdata GmbH | March 19th, 2020 S3-backed SerDe - Limitations ● Communication with S3 is costly ● Small message overhead (1 byte) ● Partitioning affected if used for keys ● Dangling references if compaction is used ● No KSQL support (KSQL Formats)
- 17. End-to-end large messages processing with Kafka Streams & Kafka Connect | Philipp Schirmer | bakdata GmbH | March 19th, 2020 Read more Processing Large Messages with Kafka Streams (Medium)
- 18. End-to-end large messages processing with Kafka Streams & Kafka Connect | Philipp Schirmer | bakdata GmbH | March 19th, 2020 Read more bakdata/kafka-s3-backed-serde (GitHub)
- 19. End-to-end large messages processing with Kafka Streams & Kafka Connect | Philipp Schirmer | bakdata GmbH | March 19th, 2020 bakdata/faust-s3-backed-serializer (GitHub) Read more
- 20. End-to-end large messages processing with Kafka Streams & Kafka Connect | Philipp Schirmer | bakdata GmbH | March 19th, 2020 Contact us [email protected] @bakdata https://www.hiclipart.com/free-transparent-background-png-clipart-qbavj https://upload.wikimedia.org/wikipedia/en/thumb/9/9f/Twitter_bird_logo_2012.svg/300px-Twitter_bird_logo_2012.svg.png