Data-Streaming at DKV
0 likes•170 views
Management for Kafka is a tool that applies GitOps to orchestrate Kafka clusters. It manages Kafka entities like topics and service accounts through structured storage and parameterizable scripts. The CLI pod for Kafka allows developers to debug securely following least privilege by deploying pods with service account credentials and automatically cleaning up.
Data-Streaming at DKV
- 1. Data-Streaming at DKV Tobias Gockel, Alexander Kropp Frankfurt, 20.10.2022 GitOps-Integration for the management of Kafka resources
- 2. Management Summary Tobias Gockel Team manager Platform Customer Product Services @ DKV Mobility Alexander Kropp IT Consultant Starting with the context of how our organization is built around cross functional product teams and how we are developing software at DKV we want to describe two solutions for working with Kafka: Management for Kafka is a tool that applies the GitOps approach1 to Kafka cluster orchestration. The CLI pod for Kafka allows developers to comfortably debug following the principle of least privilege2. 1 GitOps is an operational framework that takes DevOps best practices used for application development such as version control, collaboration, compliance, and CI/CD tooling, and applies them to infrastructure automation. 2 A subject should be given only those privileges needed for it to complete its task. 2
- 3. Data-Streaming: Organizational context Customer Product Services: CPS develops digital products in the cloud for our customers. Platform-Team Cockpit Framework Pricing and Maps Produkt-Team 1 Produkt-Team 2 Produkt-Team 3 Produkt-Team 4 Produkt-Team 5 Produkt-Team 6 Platform-Team Cloud Infrastructure CI/CD Toolchain Kafka Cluster IAM Integrated Portal Smartphone App Framework Development Product Development Customer Product Services App Framework 3 Self-enablement Scalability Security Automation Key principles:
- 4. Data-Streaming: Motivation Kafka orchestration at DKV Mobility Agile Software Development at DKV Mobility Fairly mature and modern stack and process GUI via confluent Control Center No access for developers Cumbersome workflow for creating Kafka resources - Topics - Service Accounts - ACLs1 - API-Keys No agile development and not a mature and efficient process 1 Access Control Lists Local development Remote development Dev/Test stage Test/PreProd stage Prod stage Operations 4
- 5. Management for Kafka: Technical context VNET peering Confluent Cloud DKV Azure Cloud Dev Test Pre Prod Kubernetes Cluster Kafka Connect Cluster inside Kubernetes Azure Resources Self-hosted agents Self-enablement Scalability Security Automation Key principles: Dev/Test Pre Prod Dedicated Cluster VNET peered Multi Availability Zones 5
- 6. Management for Kafka - Architecture 6 Management for Kafka Entity Management Topics Service Accounts ACLs Validation Data Schema Policies Consistency Checks Planning Desired State Current State Changes Deployment Creation Deletion Updating Secret Handling
- 7. Management for Kafka – Entity Management 7 Management for Kafka Entity Management Topics Service Accounts ACLs Validation Data Schema Policies Consistency Checks Planning Desired State Current State Changes Deployment Creation Deletion Updating Secret Handling • Structured storing of Kafka entities • Parameterized scripts to create, update or delete Kafka entities • Uses the validation component to check if changes are allowed • Parameterized pipelines to simplify the Kafka entity management • Avoid wrong inputs • Simplifies to be compliant to policies • Automated pull requests • Create multiple resources at once • Low complexity • Alternatively manual pull requests Characteristics
- 8. Management for Kafka - Validation 8 Management for Kafka Entity Management Topics Service Accounts ACLs Validation Data Schema Policies Consistency Checks Planning Desired State Current State Changes Deployment Creation Deletion Updating Secret Handling • Validates data schemes of Kafka entities • Topic config in the correct format? • Config contains only possible values and no nonsense? • Customizable policies of Kafka entities • Naming conventions • Required metadata • Config restrictions for topics or service accounts • Consistency checks • Is there a specific stage order? (e. g. topics should only exist on higher stages if they are already on lower stages) • Should a Kafka entity exist and not exist at the same time? Characteristics
- 9. Management for Kafka - Planning 9 Management for Kafka Entity Management Topics Service Accounts ACLs Validation Data Schema Policies Consistency Checks Planning Desired State Current State Changes Deployment Creation Deletion Updating Secret Handling • Combines all stored Kafka entities to create a desired state • Validation component to check if Kafka entities are valid • Topics and service accounts which should or should not exist • Topic configurations • API-Keys which should exist (supports Kafka cluster and Confluent schema registry) • ACLs which should exist • Uses API to get the current state • Compares current state with desired states • Creates a plan with changes • Does not change resources which are not part of Kafka management • Wrapped inside a parameterizable pipeline Characteristics
- 10. Management for Kafka - Deployment 10 Management for Kafka Entity Management Topics Service Accounts ACLs Validation Data Schema Policies Consistency Checks Planning Desired State Current State Changes Deployment Creation Deletion Updating Secret Handling • Deploys changes according to the plan • Outputs results • Actual created, updated or deleted Kafka entities • Stores API-Keys • Wrapped inside a parameterizable pipeline • Approvals • Stores API-Keys automatically to dedicated Azure Key Vaults Characteristics
- 11. Management for Kafka - Summary 11 Management for Kafka Entity Management Topics Service Accounts ACLs Validation Data Schema Policies Consistency Checks Planning Desired State Current State Changes Deployment Creation Deletion Updating Secret Handling Self-enablement Scalability Security Automation Key principles: Easy to use
- 12. Debugging Kafka - Motivation 12 Services access the Kafka cluster with dedicated service accounts with a limited set of ACLs Initial situation principle of least privilege Why should developers do that differently? (e. g. personal accounts with access on (almost) everything) Let us create a self-service to enable the developers to securely debug their specific Kafka resources! Our vision ☺
- 13. Debugging Kafka - Solution 13 1 2 4 3 5 Request Pod via Azure DevOps Get credentials for Service Account Deploy Debug service Clean up automatically Step description 1 2 3 4 5 Workflow
- 14. Debugging Kafka - Solution 14 • Self-service to create Kubernetes deployment to debug Kafka • Same approval rules as other deployments • Contains Kafka-CLI scripts and custom scripts • Access rights of a specific service account • Enforces that teams can only use service accounts which belong to them • Gets cleaned up automatically • Convenient and secure way to debug Kafka applications Characteristics Self-enablement Scalability Security Automation Key principles: 1 2 3 4 5 Workflow Request Pod Get credentials Deploy Debug Clean up
- 15. Data Streaming @ DKV – Wrap Up Kafka orchestration at DKV Mobility Agile Software Development at DKV Mobility Fairly mature and modern stack and process Local development Remote development Dev/Test stage Test/PreProd stage Prod stage Operations Fairly mature and modern stack and process Key principles Self-enablement Scalability Security Automation 15
- 16. 16