Putting Kafka Together with the Best of Google Cloud Platform
0 likes•2,729 views
The document discusses the integration of Apache Kafka on Google Cloud Platform (GCP) and its evolution from Google's earlier messaging systems like Google File System and Pub/Sub. It compares the scaling models and efficiency of Kafka and Cloud Pub/Sub, highlighting factors such as performance, messaging patterns, and architectural choices. The content also addresses the user flows and interactions with other GCP services, providing insights into cost and availability considerations.
![Google File System* (2003)
“Hundreds of producers [...] concurrently append to a file.”
“...files are only read, and often only sequentially”
“...files are often used as producer-consumer queues”
*”The Google File System”, Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung (Google), SOSP’03, Copyright
ACM 2003. tinyurl.com/y7r86zyc](https://image.slidesharecdn.com/07kirtitievsky-181023060608/85/Putting-Kafka-Together-with-the-Best-of-Google-Cloud-Platform-6-320.jpg)
Putting Kafka Together with the Best of Google Cloud Platform
- 1. Google Meets Apache Kafka Kir Titievsky Product Manager Google Cloud Platform, Messaging
- 3. Messages History Patterns Building Apache Kafka on GCP Patterns (Kafka on GCP)
- 4. History
- 6. Google File System* (2003) “Hundreds of producers [...] concurrently append to a file.” “...files are only read, and often only sequentially” “...files are often used as producer-consumer queues” *”The Google File System”, Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung (Google), SOSP’03, Copyright ACM 2003. tinyurl.com/y7r86zyc
- 7. Not quite Kafka Redacted: Fig 3 from the Google Filesystems Paper (Copyright ACM 2003)
- 8. pubsub 2005 pubsub1 ... is “hard to maintain” 2009 pubsub2 2015 Cloud Pub/Sub 2018 TB/s on pubsub2
- 9. M Scaling model: Kafka -- partitions Order! Stateful apps/CDC! Hot keys + head of line blocking. Consumer ConsumerM M M M M M MM M M X M M M M M M M M M M M M M
- 10. Max throughput Client/messages Stateless 😃 No order 😲 Cloud Pub/Sub Scaling model: Cloud Pub/Sub -- messages Consumer Consumer Consumer M M M M M M M M M M M M M M M M M M M M MM
- 11. Cloud Pub/Sub - Store close to publisher - Collate for subscribers Global messaging us-west FE Topic A Topic A Topic A Subscription X LB (one hostname, namespace) us-east FE eu-west US BE Subscription Y EU BE M M M M M M M M M M M M
- 12. Patterns
- 13. EAI in IT: monolith refactored Applicant Candidate Employee User Apps pubsub * Architecture approximate
- 14. Lambda lives Ads FE Ads FE Ads FE Ads FE Ads FE Ads FE pubsub Batch updater DB Config API
- 15. Ads FE Ads FE Ads FE Ads FE Extreme availability pubsub blue pubsub green pubsub black state state state Ads FE Ads FE Ads FE Ads FE Ads FE Ads FE Ads FE
- 16. But wait... what if everything else scaled? ● Kubernetes / Google App Engine ● Bigtable, Apache HBase, Apache Cassandra.... ● Spanner (transactions)
- 18. Freedom to optimize features vs. cost Apache Kafka Cloud Pub/Sub Authentication 82 4 Logging 54 0 Semantics 44 4 Performance 39 24 Kafka-Specific 78 0 297 32
- 19. ● 20MB/sec ● 0.5 days of storage ● 52TB / month published ● $0.002/GB Super-efficiency Single node Kafka cluster 4 vCPUs, 3.6 GB RAM $72/month 1TB disk (100MB/sec) $40/month
- 20. Availability is 10X the price (2-10¢/GB) Zone 2 Zone 1 Follower Zone 3 Follower 1¢/GB zone egress Producer Consumer Consumer Producer Leader Producer 2/3 1/3 1/3 2/3
- 21. Patterns from GCP users
- 22. Flows App Cloud Pub/Sub Dataflow BigQuery Kafka KSQL, Streams Kafka Connect
- 23. GCP On prem App BigQueryPub/Sub Dataflow Kafka Kafka Connect App
- 24. PCollection<String> input = p.apply("KafkaIngest1", KafkaIO.<Long, String>read() .withBootstrapServers(KAFKA_BOOTSTRAP_SERVERS) .withTopic(TOPIC_NAME) .withKeyDeserializer(LongDeserializer.class) .withValueDeserializer(StringDeserializer.class) .withoutMetadata()) // // ------- GENERIC TRANSFORMS ------- .apply("Parse", Values.create()) .apply("AddTimeStamp", ParDo.of( new DoFn<String, String>() { ... Reading from Kafka in Apache Beam
- 25. Stream=batch in processing layer vs. storage
- 26. ***@google.com Kir Titievsky, Product Manager Google Cloud Platform Thanks & Happy Streaming!