Cloud-Based Event Stream Processing Architectures and Patterns with Apache Kafka | Israel Ekpo, Microsoft
Download as PPTX, PDF0 likes850 views
The document discusses cloud-based event stream processing with Apache Kafka, focusing on deployment options, integration patterns, and best practices for success. It highlights the importance of understanding the environment, choosing between fully-managed and self-managed solutions, and the role of different architectural patterns. The conclusion emphasizes the need for the right team and clear objectives to effectively implement these technologies.
Cloud-Based Event Stream Processing Architectures and Patterns with Apache Kafka | Israel Ekpo, Microsoft
- 1. Cloud-Based Event Stream Processing Architectures and Patterns with Apache Kafka KAFKA SUMMIT EUROPE 2021 Israel Ekpo
- 3. Outline Tips for success Patterns for the Cloud Resources Follow up and Next Steps
- 7. Where, Why, How, What, Who
- 9. Where, Why, How, What, Who
- 10. Where, Why, How, What, Who
- 11. Cloud-Based Event Stream Processing with Apache Kafka
- 12. • Brokers • Schema Registry • Kafka Streams, KSQLDB • Kafka Connect Ecosystem • Kafka Client Ecosystem
- 13. • Operations, Data Engineers, Data Scientists, Solutions Architects • System Architecture, DevOps Pipeline, Agile Methodologies • Software Product or Experience
- 15. •Architectural Complexity •Operating Complexity •Lack of Trust and Desire for Control
- 16. Patterns for the Cloud
- 20. Fully-Managed on Azure Confluent Cloud Integration on Azure Azure Event Hubs
- 21. Self-Managed On Azure Confluent Platform on Virtual Machines Confluent Platform on Kubernetes (via Confluent Operator) Apache Kafka on Azure HDInsight
- 23. Hybrid Scenarios
- 25. Hybrid Scenarios for Flexibility Dynamic Capacity Needs (Kafka Streams, KSQLDB, etc) Some Components not available in Fully-Managed Offering Custom Scenarios
- 26. Stretch Clusters
- 27. Tiered Storage
- 31. Integration Options via Connect Framework Source Connectors Sink Connectors https://www.confluent.io/hub/#azure
- 32. Summary Event Stream Meshes via KSQLDB Orchestration vs Choreography Dynamic Capacity Stretch Clusters Tiered Storage
- 33. Summary Mindset Change People Deployment Options Awareness of Environment and Integration Options
- 34. Conclusion Get the right team in place Understand your environment and what is possible Good process in place Clear goals and objectives Reach out
- 36. Thank you
Editor's Notes
- #12: We need to understand the Where, Why, How, What, Who
- #32: Azure Service Bus (Source) Azure Event Hubs (Source) Azure Blob Storage (Source and Sink) Azure Cosmos DB Connector (Source and Sink) Azure IoT Hub Connector (Source and Sink) Azure Data Lake Storage Gen1 (Source and Sink) Azure Data Lake Storage Gen2 (Sink) Azure Cognitive Search (Sink) Azure Functions (Sink) Azure Synapse Analytics (Sink) Azure Log Analytics (Sink) Azure Data Explorer (Sink)