Apache Kafka Architectures and Fundamentals
Download as PPTX, PDF3 likes2,854 views
The document outlines the essential concepts and functionalities of Apache Kafka, including producer and consumer client programming logic, durability features, and security measures. It discusses partition leadership, data retention policies, exactly once semantics, and consumer group dynamics. Additionally, it highlights Kafka's encryption capabilities and invites readers to register for upcoming Confluent streaming events.
Apache Kafka Architectures and Fundamentals
- 1. Apache Kafka Architectures and Fundamentals Henrik Janzon, Solutions Engineer
- 2. Learning Objectives After this module you will be able to: ● Give a high level description of the programming logic in Kafka producer and consumer clients ● Explain how EOS works to an interested lay person ● List the means with which Kafka provides durability and HA ● Illustrate on a high level, how you can secure your Kafka cluster
- 3. Copyright 2020, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. 3
- 4. Copyright 2020, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. 4
- 5. Partition Leadership & Replication
- 6. Partition Leadership & Replication
- 7. Data Retention Policy How long do I want or can I store my data? • How long (default: one week) • Set globally or per topic • Business decision • Cost factor • Compliance factor (e.g., GDPR)
- 11. Producer Design
- 15. Exactly Once Semantics (EOS) What is it? ● Strong transactional guarantees for Kafka ● Prevents clients from processing duplicate messages ● Handles failures gracefully Use Cases ● Tracking ad views ● Processing financial transactions ● Stream processing
- 16. Partitioning Strategies Why partitioning? ● Consumers need to aggregate or join by some key ● Consumers need ordering guarantee ● Concentrate data for storage efficiency and/or indexing
- 17. Consumer Groups
- 19. Security Overview ● Kafka supports Encryption in Transit ● Kafka supports Authorization and Authentication ● No Encryption at Rest out of the box ● Clients can be mixed with & without Encryption & Authentication
- 20. Client Side Security Features ● Encryption of Data in Transit ● Client Authentication ● Client Authorization
- 21. Keen to learn more? Register for one of the Confluent Streaming Events 07 October - Middle East 12 October – Nordics 15 October – Rest of Europe Visit: https://events.confluent.io/
- 22. Q&A