Q&A with Confluent Experts: Navigating Networking in Confluent Cloud
0 likes115 views
This document discusses networking options and best practices for Confluent Cloud. It provides an overview of public endpoints, private link, and peering options. It then discusses best practices for private networking architectures on Azure using hub-and-spoke and private link designs. Finally, it addresses networking considerations and challenges for Kafka Connect managed connectors, as well as planned enhancements for DNS peering and outbound private link support.
Q&A with Confluent Experts: Navigating Networking in Confluent Cloud
- 1. Thanks for joining! We’ll get started soon! Technical Enablement Session
- 2. Partners Q&A
- 3. Partners Q&A
- 4. @yourtwitterhandle | developer.confluent.io Our Partner Technical Enablement offering Scheduled sessions On-demand Join us for these live sessions where our experts will guide you through sessions of different level and will be available to answer your questions. Some examples of sessions are below: • Confluent 101: for new starters • Hybrid Cloud Workshop: learn by doing • Path to Production series , Confluent Cloud workshops series • Product Updates Learn the basics with a guided experience, at your own pace with our learning paths on-demand. You will also find an always growing repository of more advanced presentations to dig-deeper. Some examples are below: • Aware/Novice/Competent Learning paths • Confluent Use Cases • Positioning Confluent Value • Confluent Cloud Networking • … and many more AskTheExpert we’ll offer a channel dedicated to streaming questions • Build CoE inside partners by getting people with similar interest together • Connect with opportunities and discover trends at focus partners • Build a Technical Community • Q&A • Tech Talk
- 5. @yourtwitterhandle | developer.confluent.io What are the best practices to debug client applications (producers/consumers in general but also Kafka Streams applications)?
- 7. March 19-20, 2024 ExCeL London #kafkasummit cnfl.io/KSL2024
- 8. March 19-20, 2024 ExCeL London REGISTER NOW
- 9. Partners Q&A
- 10. @yourtwitterhandle | developer.confluent.io On the board for today Confluent Cloud Networking Overview Best Practises for Private Networking Networking for Kafka Connect Ask me anything networking!
- 11. @yourtwitterhandle | developer.confluent.io Confluent Cloud Networking - Options Public Endpoints What we lead with, the classic SaaS model. Pros: • Easy, Short Time to Code, Flexible Connectivity • Confluent + CSP runs all infra • Consistent across CSPs Cons: • Might not meet all regulatory environments, we ask why, always and compare to other services. • Might not meet all regulatory environments, we
- 12. @yourtwitterhandle | developer.confluent.io Confluent Cloud Networking - Options Private Link Primary Private Networking Option Pros: • Very agreeable to regulatory security posture • Strategic private networking option for both Confluent and CSPs Cons: • Its Private, requires additional networking; connectivity, routing, security, DNS…. all managed by the customer, drives OPEX/CAPEX • External access challenges • CSP specific caveats/limits are inherited
- 13. @yourtwitterhandle | developer.confluent.io Confluent Cloud Networking - Options Peering Legacy Private Networking Pros: • Easy starting point for Private Networking Cons: • Its Private, non-Transitive & requires additional networking • Confluent is part of customer network, security concerns • External access challenges • CSP specific caveats/limits are inherited
- 14. @yourtwitterhandle | developer.confluent.io Confluent Cloud Networking - Options Transit Gateway (AWS Only) Large Scale Private Networking Pros: • Scales for regional, global and cross CSP environments. Cons: • Its Private, requires additional networking; connectivity, routing, security…. all managed by the customer, drives OPEX/CAPEX • Confluent is part of customer network, security concerns • External access challenges • AWS specific
- 15. Best Practises for Private Networking Go-to Architecture when Public Endpoints are not accepted
- 16. FW FW Peering Hub VNet DMZ & Landing Zone VNet Private Link Endpoint(s) Private Link Services Private Zone(s) for PL Endpoints Confluent Azure Tenant Customer Azure Tenant FW FW Peering Peerings Hub VNet DMZ VNet Private Link Endpoint(s) Private Link Service Private Zone(s) for PL Endpoints Confluent Azure Tenant Customer Azure Tenant Kafka Connect LandingZone VNet Connect VNet Kafka Connect Outbound Private Link For Connect DB Outbound Private Link For Connect DB Private Networking Best Practices - Private Link Architecture
- 17. Azure Use Case: Hub-n-Spoke Reference Architecture
- 18. Private Networking Best Practices - Azure Hub and Spoke -Peering
- 19. Private Networking Best Practices - Azure Hub and Spoke - Private Link
- 20. FW FW Peering Peering Hub VNet DMZ VNet Confluent Azure Tenant /16 Customer Azure Tenant Customer Azure Tenant UDR UDR Public DNS (Confluent Managed)
- 21. Private Networking for Managed Connectors
- 22. Copyright 2020, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. The plumbing, the foundational requirement. ● Internet ● Peering ● Transit Gateway ● Private Link ● OnPremise, Remote Networks, Multi-Cloud Friction - Connectivity Network Connectivity DNS Connector Configuration
- 23. Copyright 2020, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. If FQDNs are required and we can’t resolve, we are dead in the water. ● Public DNS ○ Public record can have a private or a public IP. ● Private DNS - Hosted Zone Requirement ● Confluent Cloud resolves DNS in Confluent VPC/VNet, private DNS zones not exposed as configurable to customers. Friction - DNS 23 Network Connectivity DNS Connector Configuration
- 24. Copyright 2020, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. ● Connector Config Options are Limited ○ Limits the use of custom endpoints ■ For example, you supply only the bucket name and the standard public endpoint is used ● storage.googleapis.com not a custom endpoint SERVICE-ENDPOINT.p.googleapis.com Friction - Connector Configuration 24 Network Connectivity DNS Connector Configuration
- 25. Enhancements in Flight - DNS Peering for TGW/Peered Clusters Q1 - AWS/Azure Q2 - GCP 25
- 26. 2 1 4 5 3 Enhancements in Flight - DNS Peering for TGW/Peered Clusters
- 27. FW FW Peering Hub VNet DMZ & Landing Zone VNet Private Link Endpoint(s) Private Link Services Private Zone(s) for PL Endpoints Confluent Azure Tenant Customer Azure Tenant Kafka Connect Outbound Private Link For Connect DB Enhancements in Flight - Outbound PL for Managed Connectors BYOC support beyond 2024. 1. Customer creates PrivateLink Service for their source/sink (like a DB). 2. Customer creates an endpoint in Confluent Cloud VPC/VNet. 3. Customer creates a DNS record in Confluent Cloud to proper resolve. Creates a clean secure solution for Managed Connectors in PL Environments (Peeering/TGW late 2024) Enables Private & Public outbound access for Managed Connectors