An open source, scalable queuing solution on top of apache kafka 2019
0 likes75 views
The document discusses implementing a scalable queuing system using Apache Kafka for a large online shopping platform with numerous products and reviews. It outlines the architecture and services involved, including the challenges of scalability, monitoring, and recoverability, as well as the integration of NLP for automatic review tagging. Additionally, it highlights the benefits of using Kafka and message queues, while addressing potential issues such as data loss and API awareness.
An open source, scalable queuing solution on top of apache kafka 2019
- 1. Implementing a scalable queuing system using Apache Kafka Yaniv Bronhaim Senior Software Engineer
- 2. Online Shopping World ● 10B total products all together ● 50K variants per product ● 50 reviews per minute
- 3. Session outline ● Services & Communication from web Point-Of-View ● Challenges ● Message Queue ● Kafka ● Innovation by Yotpo
- 4. Services & Communication CreateReview(payload) GetReviews(store_id) GetReviewByTag(tag_id) Yotpo Service
- 5. Save review with related info (user data, product data, score) Review Service CreateReview(payload) Yotpo Service Create Review Flow
- 6. Save review with related info (user data, product data, score) Review Service CreateReview(payload) Yotpo Service Insights processing (NLP algorithm that automatically tags the review) Reviews Tagging Service Create Review Flow
- 7. Save review with related info (user data, product data, score) Review Service CreateReview(payload) Insights processing (NLP algorithm that automatically tags the review) Reviews Tagging Service Challenges Yotpo Service
- 9. Save review with related info (user data, product data, score) Review Service Insights processing (NLP algorithm that automatically tags the review) Reviews Tagging Service Scalability Yotpo Service CreateReview(payload)
- 10. Save review with related info (user data, product data, score) Review Service Insights processing (NLP algorithm that automatically tags the review) Reviews Tagging Service Recoverable Yotpo Service CreateReview(payload)
- 11. Save review with related info (user data, product data, score) Review Service Insights processing (NLP algorithm that automatically tags the review) Reviews Tagging Service Monitoring Yotpo Service CreateReview(payload)
- 13. Save review with related info (user data, product data, score) Review Service CreateReview(payload) Insights processing (NLP algorithm that automatically tags the review) Reviews Tagging Service Message Queue Yotpo Service
- 14. Permits queuing of jobs for subsequent processing (efficient) The Good Using Message-Queue Can scale up (scalable)
- 15. Easy monitors and tools (intuitive) The Good Using Message-Queue Safe from data loss (recoverable)
- 16. The Bad Using Message-Queue API awareness issues Expensive (queue for each service)
- 17. Kafka Overview
- 23. Save review with related info (user data, product data, score) Review Service Yotpo Service CreateReview(payload) Insights processing (NLP algorithm that automatically tags the review) Reviews Tagging Service review_created_topic Using Kafka
- 24. Permits queuing of jobs for subsequent processing (efficient) The Good Kafka Scalable Easy monitors and tools (intuitive) One publish triggers all (centralized)
- 25. Not safe from data loss The Bad Using Kafka One point of failure
- 26. Save review with related info (user data, product data, score) Review Service Yotpo Service CreateReview(payload) Kafka + Message Queue Insights processing (NLP algorithm that automatically tags the review) Reviews Tagging Service review_created_topic
- 27. Zoom-in to Service Scale
- 28. Solution on Top of Kafka Fast Commit Requires immediate commit offset -> to allow parallel consuming from same partition Requires failure flow to allow re-producing in case of consuming error
- 29. Align Usage And Configurations Available in Go, Ruby and Java Adds Kafka-Retry-Service (KRS) Integration Yotpo Kafka Open-Source Kafka Packages
- 30. On Top of Kafka KRS Solution
- 31. On Top of Kafka KRS Solution
- 32. Payloads are Saved, Re-triggered and Monitored KRS
- 33. Summary ● Soon Available On Github ● Much More To Contribute ● On Production Since May 2019 ● I’m Available For Questions
- 34. Questions?