Uploaded byconfluent
Kafka Summit SF 2017 - Real-Time Document Rankings with Kafka Streams
Hunter Kelly presents an approach for using Apache Kafka Streams to perform real-time domain ranking based on a modified HITS algorithm. The system discovers relevant fashion domains from web links. It represents domains as a graph and runs HITS iterations to identify hub and authority domains. By using Kafka Streams, the rankings can be updated continuously in real-time from a stream of new links. The system decomposes the HITS algorithm into separate Kafka streams processes for link extraction, domain reduction, and scoring domains.
More Related Content
Similar to Kafka Summit SF 2017 - Real-Time Document Rankings with Kafka Streams
![Building streaming data applications using Kafka*[Connect + Core + Streams] b...](https://cdn.slidesharecdn.com/ss_thumbnails/buildingstreamingdataapplicationsusingapachekafka-171011211455-thumbnail.jpg?width=640&height=640&fit=bounds)
Kafka Summit SF 2017 - Real-Time Document Rankings with Kafka Streams
- 1. Hunter Kelly @retnuh Real-Time DomainRankings with Kafka Streams
- 2. What are wedoing? ● Discovering relevant domains in the Fashion Web ● Use modified HITS - a.k.a. Hubs & Authorities Why use Kafka Streams? ● Why not Spark, Hadoop, Whatevs? ● Advantages of Kafka Streams Show me the nitty gritty! ● Okay!
- 3.
- 4. Curated list ofFashion related sites ● Initial lists from domain experts ● Support multiple languages ● What are the next “best” domains?
- 5.
- 6. http://www.cs.cornell.edu/home/kleinber/auth.pdf http://en.wikipedia.org/wiki/HITS_algorithm * Hyperlink-Induced TopicSearch, but no-one seems to call it that Need Data-driven method ● Kleinberg’s HITS* algorithm ● Allows us to discover Hubs and Authorities ● Flattened to domain level
- 7. HITS in anutshell
- 8. HITS in anutshell Adjacency Matrix Image courtesy of http://faculty.ycp.edu/~dbabcock/PastCourses/cs360/lectures/lecture15.html
- 9. HITS in anutshell ● Why not just counts? ● If you point to good sites, you’re a better Hub ● If good sites point to you, you’re a better Authority ● Each iteration, the weights change, until they reach convergence
- 10. Common Questions ● Whatabout non-fashion domains? ● Why not PageRank?
- 11.
- 12. Please tell meyou aren’t using Kafka to do Matrix multiplication!
- 13. Please tell meyou aren’t using Kafka to do Matrix multiplication! No, I am not crazy enough* to use Kafka to do matrix multiplication!
- 14. Please tell meyou aren’t using Kafka to do Matrix multiplication! No, I am not crazy enough* to use Kafka to do matrix multiplication! *Although, I probably did spend too much time thinking about it on the bus!
- 15. Why not useMap-Reduce? Basically what it was invented for, right?
- 16. You could useHadoop...
- 17. You could useHadoop...
- 18. You could useSpark… ● High infrastructure overhead if not using it for anything else ● Bad initial experience w/ Spark Streaming snapshotting and recovery ● Already using Kafka!
- 19. Why use KafkaStreams? ● Has primitives necessary for Map-Reduce ○ Map step groupBy groupByKey ○ Reduce step reduce aggregate ● Focus is on your data not distributed computing machinery ● Streaming allows us to have (near) real-time, up-to-date data
- 20.
- 21. Kafka Streams 101 ●No explicit consumers/producers - plumbing handled for you ● Topics still a fundamental communication piece ● Think of individual datum flowing through KStreams & KTables
- 22. Kafka Streams 101- KSTreams ● Focus is on specific functional transformations - map, filter, flatMap ● Also supports various flavours of joins with other KStreams ● Usually created from one or more topics or a transformation on another KStream
- 23. Kafka Streams 101- KTable ● Still offers functional transforms, but on a primary-keyed table ● Offers persistent storage ● Created from aggregations on a KStream or transforms on other KTable
- 24. ● The bridgebetween KStream and KTable ● Created by doing a groupBy or groupByKey on a KStream ● Create KTables by doing reduce, aggregate, count Kafka Streams 101 - KGroupedStream
- 25. Output Topic (log compacted) KTableops: groupByKey, reduce, toStream KStream Input Topic KStream ops: flatMapDomain Link Extractor Domain Reducer HITS Calculator & API Data Flow
- 26. Output Topic (log compacted) KTableops: groupByKey, reduce, toStream KStream Input Topic KStream ops: flatMapDomain Link Extractor Domain Reducer HITS Calculator & API
- 27. Output Topic (log compacted) KTableops: groupByKey, reduce, toStream KStream Input Topic KStream ops: flatMapDomain Link Extractor Domain Reducer HITS Calculator & API
- 28. Output Topic (log compacted) KTableops: groupByKey, reduce, toStream KStream Input Topic KStream ops: flatMapDomain Link Extractor Domain Reducer HITS Calculator & API
- 29. Current Domain Link Extractor InputTopic Fetch Content External Links Resolve Links Extract Domains KStream KStream ops: flatMap
- 30. Extracted Link Resolver Domain LinkExtractor Input Topic Fetch Content External Links Resolve Links Extract Domains Output Topic KStream ops: flatMap KStream ops: flatMap KStream ops: flatMap
- 31. Domain Link Extractor InputTopic Fetch Content External Links Resolve Links Extract Domains KStream Other Function(s) KStream KStream ops: flatMap KStream Topic Whatev’s With Fork
- 32.
- 33. TL;DR ● Kafka Streamscan help solve problems in your application domain ● Focus on your data! ● Naturally decompose the problem into flexible microservices
- 35.