Kafka, the "DialTone for Data": Building a self-service, scalable, streaming analytics system @ HomeAway, Rene Parra
Download as PPTX, PDF3 likes1,570 views
Kafka is used as a "dial tone for data" to ingest large amounts of data at HomeAway. An experiment using Kafka and Camus to ingest over 1TB of data per day from various systems and applications was successful. It allowed for various use cases like SLA reporting, fraud detection, search and clickstream analysis, and traveler segmentation. Key lessons included the importance of the schema registry for decoupling producers and consumers, and making stream processing easy through tools like Samza. The presentation concludes that Kafka allows building systems of engagement and intelligence on top of ingested data.
![Lesson #1: The Schema [registry] is Everything!
Data Lake
© Copyright 2016 HomeAway, Inc.
Schema
Registry
• Decouples producers from
consumers
• Enforces backwards
compatibility
• Enables self-service /
democratization
• SOT for schemas in the pipe](https://image.slidesharecdn.com/kafkadialtonefordatareneparra-160511213434/85/Kafka-the-DialTone-for-Data-Building-a-self-service-scalable-streaming-analytics-system-HomeAway-Rene-Parra-27-320.jpg)
Kafka, the "DialTone for Data": Building a self-service, scalable, streaming analytics system @ HomeAway, Rene Parra
- 1. © Copyright 2016 HomeAway, Inc. Kafka: The “Dial Tone” for Data
- 2. HomeAway The world leader for vacation rentals > 1 million listings (and growing!)
- 3. Agenda © Copyright 2016 HomeAway, Inc. • Overview • The Problem • The Experiment • Results: Use Cases • Lessons Learned • Next Steps
- 4. © Copyright 2016 HomeAway, Inc. Overview
- 5. Difference between Dinosaurs and Unicorns © Copyright 2016 HomeAway, Inc.
- 6. In the old days: “Dial Tone” looked like this © Copyright 2016 HomeAway, Inc. ATDT
- 7. Today: Kafka is the modern “Dial Tone” for Data © Copyright 2016 HomeAway, Inc. Producer Consumer
- 8. The Problem © Copyright 2016 HomeAway, Inc.
- 9. The Problem © Copyright 2016 HomeAway, Inc.
- 10. Our original problem/motivation © Copyright 2016 HomeAway, Inc. search head indexer indexer app server forwarder app server forwarder 1 TB/day ingress and growing! 40,000 calls/sec
- 11. Also… Historical Analytic Pipeline was slow/expensive © Copyright 2016 HomeAway, Inc. app server OLTP OLAP analytics ETL
- 12. Fill the Lake! Alternatives ? Problem: Fill Hadoop! Problem Data Lake © Copyright 2016 HomeAway, Inc.
- 13. What we wanted… the Big Idea © Copyright 2016 HomeAway, Inc. If you can log it… … you can analyze it!
- 14. How to build self-service? © Copyright 2016 HomeAway, Inc.
- 15. Hypothesis: Use Kafka! © Copyright 2016 HomeAway, Inc. 2 ms median latency http://bit.ly/jay_on_logs the log 2 Million Events / Sec! (3 cheap machines) http://goo.gl/pv5GoL “Benchmarking Apache Kafka”
- 16. © Copyright 2016 HomeAway, Inc. The Experiment
- 17. HACommonsLogging • KafkaAppender Schema-on-read • KafkaAvroLogger Schema-on-write Experiment: Schema-on-Read, Schema-on-Write Data Lake © Copyright 2016 HomeAway, Inc. Schema Registry
- 18. Architecture: Kafka + Camus = BigData Ingress © Copyright 2016 HomeAway, Inc. Camus
- 19. © Copyright 2016 HomeAway, Inc. The Results
- 20. Use Cases: ITOA / SLA Reporting © Copyright 2016 HomeAway, Inc.
- 21. Use Cases: ITOA / SLA Reporting © Copyright 2016 HomeAway, Inc.
- 22. Use Cases: Fraud © Copyright 2016 HomeAway, Inc.
- 23. Use Cases: Search + ClickStream © Copyright 2016 HomeAway, Inc. User Behavior Search Requests A/B Test Readouts Proctor EDAP
- 24. Use Cases: Search + ClickStream © Copyright 2016 HomeAway, Inc.
- 25. Use Cases: Traveler Segmentation © Copyright 2016 HomeAway, Inc. EDAP Data Model
- 26. Lessons Learned © Copyright 2016 HomeAway, Inc.
- 27. Lesson #1: The Schema [registry] is Everything! Data Lake © Copyright 2016 HomeAway, Inc. Schema Registry • Decouples producers from consumers • Enforces backwards compatibility • Enables self-service / democratization • SOT for schemas in the pipe
- 28. Lesson #2: A Kafka/SR governance module is helpful Data Lake © Copyright 2016 HomeAway, Inc. • TURN OFF Auto Topic Creation! • Need a place for developers to request topics • Retention Policy • Expected Load • Compaction • Partition Size / Partition Key • Owner • LTS Date
- 29. Lesson #3: Make it easy to do stream processing © Copyright 2016 HomeAway, Inc. Schema Registry • samza-archetype • samza-job-deployer • Will evaluate k-streams!!!! http://www.confluent.io/blog/introducing-kafka-streams-stream-processing-made-simple
- 30. Next Steps © Copyright 2016 HomeAway, Inc.
- 31. Consistency : 3 types of Data © Copyright 2016 HomeAway, Inc. Event Document Transactional
- 32. Kafka Producer Spooling © Copyright 2016 HomeAway, Inc.
- 33. Conclusion © Copyright 2016 HomeAway, Inc.
- 34. Yesterday © Copyright 2016 HomeAway, Inc. Systems of Record
- 35. Today © Copyright 2016 HomeAway, Inc. Systems of Engagement
- 36. Tomorrow © Copyright 2016 HomeAway, Inc. Systems of Intelligence
- 37. Don’t be a dinosaur… © Copyright 2016 HomeAway, Inc. ATDT
- 38. Thank you © Copyright 2016 HomeAway, Inc.
- 39. End of Presentation © Copyright 2016 HomeAway, Inc.