Best Practices for Building Hybrid-Cloud Architectures | Hans Jespersen
Download as PPTX, PDF•1 like•421 views
The document discusses best practices for building hybrid-cloud architectures with a focus on Confluent's Kafka solutions. It outlines the importance of cloud migration and the use of Confluent Replicator for data replication and integration across different data centers and cloud environments. Additionally, it introduces Confluent Cloud as a platform that simplifies Kafka deployment and management while providing high performance and reliability.
Best Practices for Building Hybrid-Cloud Architectures | Hans Jespersen
- 1. 1 Best Practices for Building Hybrid-Cloud Architectures Hans Jespersen VP, WW Systems Engineering Confluent Inc PARIS - 11 OCTOBER 2018
- 3. 3 Agenda The Great “Cloud Shift” What is a Bridge to Cloud? What is Confluent Cloud? Customer Stories Q&A
- 4. 4 The Great “Cloud Shift” A cloud product is not complete without a cloud migration story.
- 5. 5 Cloud Migration: A one time thing?
- 6. 6 In reality, we keep running Our apps are complex, intricately tied together. Some, built to run on legacy, immovable systems
- 7. 7 We don’t want to just move. We want to build for the cloud.
- 8. 8 DC-1 App App App App App App Elasticsearch MySQL Oracle Teradata NoSQL Database Relational Database Data Warehouse Blob Storage
- 9. 9 DC-1 App App App App App App Elasticsearch MySQL Oracle Teradata NoSQL Database Relational Database Data Warehouse Blob Storage DC-NY DC-LON amer-northeast emea-central
- 11. 11 What is our bridge? Confluent Replicator
- 12. 12 Establish Your Foundation App Oracle NoSQL Database Blob Storage 1 Deploy Kafka on-premise and your cloud provider of choice, Teradata Confluent Replicator
- 13. 13 Establish Your Foundation App Oracle NoSQL Database Blob Storage 2 Create your pipeline and replicate your topics to your cloud cluster Teradata Confluent Replicator
- 14. 14 Let the Traffic Flow App Oracle NoSQL Database Blob Storage 3 Migrate app by app, database by database App Topic: dc1users Topic: dc1clicks
- 15. 15 DC-1 App App App App App App Elasticsearch MySQL Oracle Teradata NoSQL Database Relational Database Data Warehouse Blob Storage DC-NY DC-LON amer-northeast emea-central
- 16. 16 Confluent Replicator vs MirrorMaker Feature Benefit MirrorMaker Replicator Data replication Real-time event streaming between Kafka clusters and data centers Yes Yes Schema replication Integrate with Confluent Schema Registry for multi- dc data quality and governance Yes Yes Connect replication Manage data integration across multiple data centers Yes Yes Flexible topic selection Select topics with white-lists, black-lists and regular expressions Partial* Yes Auto-create topics New topics are automatically detected and replicated Partial* Yes Add new partitions New partitions are automatically detected and replicated No Yes Configuration replication Identical topic configuration between the two clusters No Yes Single Message Transformations Filter, modify and route events on the fly No Yes Auto-scale Scale replication processes as Kafka traffic increases with a single configuration. No Yes Active-active replication Redirect events to avoid infinite replication loops in active-active configurations No Yes Aggregate cluster Replicate from multiple clusters into one (aggregator / hub-and-spoke pattern). One management point for replicating more than a single cluster. No Yes End-to-end monitoring and management Manage and monitor replication via Control Center UI No Yes License Apache Proprietary https://docs.confluent.io/current/multi-dc/mirrormaker.html
- 17. 17 Multi-Cloud, DR and Bridge to Cloud Enables multi-DC deployments of Apache Kafka ● On-premises: Datacenter to Datacenter ● Hybrid: On-Premise to Cloud ● Cloud: Cloud Region to Cloud Region ● Multi-Cloud: Cloud Provider to Cloud Provider Confluent Replicator
- 18. 18 Confluent Platform as a service
- 19. 19 The most popular platform Publish and subscribe, store and process streams of data based on Kafka’s latest release Develop faster with the leading streaming platform 19 Fine tune for scale Custom partitioning Log compaction Highest throughput Best performance of any streaming data service Up to 1GB/s write, 2GB/s read throughput Retain and replay more data Up to infinite retention
- 20. 20
- 21. 21 Introducing Confluent CloudTM A Way to Liberate Developers. Confidential
- 22. 22 01 Freedom to deploy Kafka in any public cloud 22 02 Access the rich Kafka ecosystem 03 Develop faster using the leading streaming platform 04 Liberate yourself from the operations burden Introducing Confluent Cloud™
- 23. 23 ● Built on open source Apache Kafka Confluent Cloud™ Gives You Vendor Independence. * support for AWS and GCP available today ● Port your existing pipelines regardless of initial starting point ● No proprietary technology so not tied to a single provider*
- 24. 24 Leading open source streaming platform Access the Rich Kafka Ecosystem Integrates with leading data systems and apps Multi-Language Support 24 Includes Kafka Core + Streams and Connect API’s Vibrant and growing open source community Support for leading languages Bring more applications to the platform 200+ pre-built connectors
- 25. 25 Fastest Way to Develop Code is to not code at all
- 26. 26 4 ways Confluent Cloud minimizes your operations burden No Infrastructure Setup No Cluster Tuning No Managing Upgrades Deploy Kafka in Minutes Liberate yourself from the operations burden 26
- 27. 27 Feature Benefit Apache Kafka Confluent Open Source Confluent Enterprise Confluent Cloud™ Apache Kafka High throughput, low latency, high availability, secure distributed streaming platform Kafka Connect API Advanced API for connecting external sources/destinations into Kafka Kafka Streams API Simple library that enables streaming application development within the Kafka framework Additional Clients Supports non-Java clients; C, C++, Python, .NET and several others REST Proxy Provides universal access to Kafka from any network connected device via HTTP Schema Registry Central registry for the format of Kafka data – guarantees all data is always consumable Pre-Built Connectors HDFS, JDBC, Elasticsearch, Amazon S3 and other connectors fully certified and supported by Confluent JMS Client Support for legacy Java Message Service (JMS) applications consuming and producing directly from Kafka Confluent Control Center Enables easy connector management, monitoring and alerting for a Kafka cluster Cloud Management Console Simply and effectively manage a Confluent Cloud cluster N/A Auto Data Balancer Rebalancing data across cluster to remove bottlenecks Automatic Replicator Multi-datacenter replication simplifies and automates MDC Kafka clusters Support Enterprise class support to keep your Kafka environment running at top performance Community Community 24x7x365 24x7x365 Confluent Cloud: Part of the Confluent Family
- 28. 28 ● New project ● Low budget ● Not business-critical ● Wants fast / frictionless access ● Production apps ● Budget available ● Business-critical ● Wants reliable service levels Developer VP Engineering Cloud Professional & Cloud Enterprise Cloud Professional Cloud Enterprise Unleash developer velocity Streaming service for all your enterprise applications
- 29. 29 Use Cases: from Born Digital to Brick and Mortar Companies Retail (Click Stream Analytics, Recommendations, Fraud Detection, Loyalty Apps) AdTech (Mobile In App Advertising) Online Gaming (Betting Engines, Regulatory Compliance) FinTech (Insurance, Peer to Peer Lending, Loan Processing) Log Aggregation (Application Logging, Syslog, CyberSecurity/SIEM) IoT (Sensor data ingestion, Connected Factory, Connected Car, Predictive Maintenance) Micro-Services (State Persistence, Streaming Micro Services) Legacy Middleware Modernization (Real-time ETL, MQ/JMS replacement, Mainframe Offload) Anything you can do with Kafka you can do (quicker/easier) with Confluent Cloud!
- 30. 30 Q&A
- 31. 31 ● Hosted service that reduces the burden on your operations team ● No Vendor Lock In - take all your streaming applications to another Kafka provider ● Get Visibility - Operations Dashboard gives you full transparency into your Kafka pipelines - by topic, by cluster, by app ● Supported by the team who created Kafka and has the most experience working in large scale streaming Confluent Cloud™ for the Enterprise: Scale without the Staff 31