Domain Driven Data: Apache Kafka® and the Data Mesh
2 likes407 views
The document discusses the integration of data mesh concepts within Apache Kafka, highlighting the challenges of monolithic data storage and centralized processing. It emphasizes treating data as a product by breaking it into domain-oriented structures, similar to microservices, while Kafka serves as a central hub for facilitating data-driven operations. Key features such as ksqlDB and schema registry support real-time processing and data governance within this framework.
Domain Driven Data: Apache Kafka® and the Data Mesh
- 1. Domain Driven Data Apache Kafka and the Data Mesh James Gollan, Senior Solutions Engineer at Confluent
- 2. Why are we talking about Data Mesh at a Kafka meetup?
- 3. Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Producer Guarantees P Broker 1 Broker 2 Broker 3 Topic1 partition1 Leader Follower Topic1 partition1 Topic1 partition1 Producer Properties acks=all min.insync.replica=2 ack
- 4. Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. What problem are we trying to solve? 4 Monolithic datastores Centralised processing and governance Bottlenecks for processing and analysing data for the business Data scientists don’t have a full understanding of the data’s context Results in a data swamp
- 5. Data mesh is an attempt to solve these problems
- 6. Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Data Mesh 6 Concept first spoken about by Zhamak Dehghani from ThoughtWorks Break apart the ‘data monolith’ Treats domains as first class citizens when dealing with data Domains encouraged to stop treating data as an asset, and to start treating data as a product Emphasis on a self-service data platform Federated governance of organisational data
- 7. This seems kinda familiar - where have I heard this before.... Could it be microservices?
- 8. Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Microservices and Domain Driven Design 8 This problem has been solved before for the Monolithic application This was broken down into microservices Creation of these microservices emphasises business domains The bounded context provides the public interfaces for the domain Within the bounded context domain specific language and business logic is used
- 10. Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. 10
- 11. Summing up
- 12. Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Kafka’s role in the data mesh Kafka Kafka facilitates the data mesh by acting as a central hub for events Infinite storage in Kafka allows it to be used as the source of truth within the organization 12
- 13. Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Kafka’s role in the data mesh Connect Domain based connect workers allow domain owners to integrate source and sink connectors One of these sinks might be a data warehouse where the domain can conduct analysis on their data sets, potentially combined with organisational wide data sets from other domains 13
- 14. Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Kafka’s role in the data mesh ksqlDB Distributed ksqlDB allows domains to run their own real-time stream processing This may be used to prepare data from multiple topics for publication across the organisation It may also be used for advanced stream processing, such as real-time fraud detection 14
- 15. Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Kafka’s role in the data mesh Schema registry With event driven architecture the schema is the API Schema registry ensures consistency in event structure, and enables forward and backward compatibility across schema changes Is may be extended to provide more data governance features, such as field level tagging, data catalog functionality etc. 15
- 16. Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Converters
- 17. Questions (or did we run out of time?)