GCP for Apache Kafka® Users: Stream Ingestion and Processing
2 likes2,748 views
The document discusses the integration of Apache Kafka with Google Cloud for stream ingestion and processing, highlighting the benefits of using Google Cloud's managed Kafka service. It covers key features like Apache Beam, Dataflow, and the ease of setting up a Kafka cluster in GCP. Additionally, it emphasizes Kafka's role in hybrid cloud environments and offers a promotional discount for related services.
![Credit
Applications
{
"ApplicationId": "997c7045-3497-4aba-b7fe-8ff48d4e8dd3",
"FirstName": "Felipe",
"LastName": "Jast",
"CreditScore": 644,
"IPAddress": "64.73.63.94",
"Company": “Company-Jast”,
"EmailAddress": "Felipe.Jast@Company-Jast.com",
"TimestampsAsJSON": "["2018-09-24T17:57:58.2971524Z"]"
}](https://image.slidesharecdn.com/slides-gcpforapachekafkausersstreamingestionandprocessing-190521200722/85/GCP-for-Apache-Kafka-Users-Stream-Ingestion-and-Processing-12-320.jpg)
GCP for Apache Kafka® Users: Stream Ingestion and Processing
- 1. GCP for Apache Kafka® Users: Stream Ingestion and Processing Karthi Thyagarajan, Solutions Architect, Google Cloud Ricardo Ferreira, Developer Advocate, Confluent (@riferrei)
- 2. Speakers Karthi Thyagarajan Solutions Architect, Google Cloud Ricardo Ferreira Developer Advocate, Confluent
- 3. Confluent Wins Google Cloud Technology Partner of the Year Award for Data and Analytics
- 4. GCP’s architecture for Stream Analytics Endpoint clients User & device data Or Or Ingest Transform Analyze (data warehouse) Web IoT Mobile PubSub Apache Kafka Apache Beam Dataflow KSQL BigQuery ML BigTable Data Studio 3rd-party BI Tools Data consumers
- 5. Components Ingestion & delivery Df KS S PS K Df KC S Cloud Pub/Sub Apache Kafka Processing Apache Beam / Dataflow Kafka Streams / KSQL Apache Spark / Dataproc Connectors Apache Beam IOs Kafka Connect Apache Spark / Dataproc
- 6. What is Beam and Dataflow? Open source programming model Unified batch and streaming Exactly-once streaming semantics Cloud Dataflow Uses Apache Beam as its SDK Automatic optimizations scale to millions of QPS Serverless, fully managed data processing
- 7. Apache Kafka as a Service for Google Cloud
- 8. Apache Kafka as a Service for Google Cloud Clients Connectors Management More than simply a managed cluster service!
- 9. Demo: Creating a Kafka Cluster in GCP in Less Than 3 Minutes using
- 10. Demo Processing
- 11. Demo: Join Kafka & Pub/Sub Topics On-Premise Cloud Dataflow Orange arrow indicates Cloud Dataflow PS BQ Cloud App CCE cluster in a peered VPC On-Prem App DB KC
- 12. Credit Applications { "ApplicationId": "997c7045-3497-4aba-b7fe-8ff48d4e8dd3", "FirstName": "Felipe", "LastName": "Jast", "CreditScore": 644, "IPAddress": "64.73.63.94", "Company": “Company-Jast”, "EmailAddress": "[email protected]", "TimestampsAsJSON": "["2018-09-24T17:57:58.2971524Z"]" }
- 13. Demo: Showing how to process stream of events using Schema Registry and KSQL Ricardo
- 14. Demo take action
- 15. Demo: Showing how to send the results of a processing stream to Cloud Functions using Kafka Connect
- 16. Kafka Best Practices for Hybrid Cloud / Cloud Native Hybrid Cloud Cloud-Native On-Premises Network Applications Network (VPC) Producer Consumer Producer Consumer Replicator Kafka Producer Kafka Consumer VPC Peering Kafka Cluster Network (VPC)
- 17. Confluent is Available Everywhere! BigQuery Colocated DC Legacy app Public Cloud Another app Messaging Apache Kafka Kafka tools, skills, replication infrastructure Messaging Apache Kafka Messaging Apache Kafka
- 18. Try Apache Kafka as a Service for GCP Today! https://cnfl.io/next19
- 19. Code: KS19Online25 25% DISCOUNT* *Standard Priced Conference pass
- 20. Thank you