Stream Processing in the Cloud - Athens Kubernetes Meetup 16.07.2019
- 1. 1 Stream Processing in the Cloud Rafał Leszko (@RafalLeszko) Cloud Software Engineer at Hazelcast
- 2. Hands Up
- 3. Hands Up Raise your hand if… ● ...you know what Stream Processing is?
- 4. Hands Up Raise your hand if… ● ...you know what Stream Processing is? ● ...you have ever used Stream Processing?
- 5. Hands Up Raise your hand if… ● ...you know what Stream Processing is? ● ...you have ever used Stream Processing? ● ...you have ever used Hazelcast Jet?
- 6. Agenda ● Part 1: Stream Processing Basics ○ What is Stream Processing and Hazelcast Jet? ○ Example: Word Count ● Part 2: Jet Under the Hood ○ How does it work? ○ Infinite Streams ○ Example: Twitter Cryptocurrency Analysis ● Part 3: Jet in the Cloud ○ Cloud (Kubernetes) integration ○ Example: Stock Trade Aggregator ● Part 4: Jet Features & Use Cases ○ Why would I need it? ○ Example: Web Crawler
- 7. Part 1: Stream Processing Basics
- 11. What is Hazelcast Jet?
- 12. What is Hazelcast Jet? DAG - Direct Acyclic Graph
- 13. What is Hazelcast Jet?
- 14. What is Hazelcast Jet?
- 15. Example 1: Word Count Problem: Count the number of occurrences of each word in the given text. Sample Input: Lorem ipsum dolor, dolor. Sample Output: lorem=1 ipsum=1 dolor=2
- 16. Example 1: Word Count Pure Java Pattern delimiter = Pattern.compile("W+"); return lines.entrySet().stream() .map(e -> e.getValue().toLowerCase()) .flatMap(t -> Arrays.stream(delimiter.split(t))) .filter(word -> !word.isEmpty()) .collect( groupingBy( identity(), counting()));
- 17. Example 1: Word Count
- 18. Example 1: Word Count
- 19. Example 1: Word Count
- 20. Example 1: Word Count
- 21. Example 1: Word Count
- 22. Example 1: Word Count Hazelcast Jet Pattern delimiter = Pattern.compile("W+"); Pipeline pipeline = Pipeline.create(); pipeline.drawFrom(Sources.<Long, String>map(LINES)) .map(e -> e.getValue().toLowerCase()) .flatMap(t -> traverseArray(delimiter.split(t))) .filter(word -> !word.isEmpty()) .groupingKey(wholeItem()) .aggregate(counting()) .drainTo(Sinks.map(COUNTS)); return pipeline;
- 23. Example 1: Word Count Pure Java Pattern delimiter = Pattern.compile("W+"); return lines.entrySet().stream() .map(e -> e.getValue().toLowerCase()) .flatMap(t -> Arrays.stream(delimiter.split(t))) .filter(word -> !word.isEmpty()) .collect( groupingBy( identity(), counting()));
- 24. Example 1: Word Count Hazelcast Jet Pattern delimiter = Pattern.compile("W+"); Pipeline pipeline = Pipeline.create(); pipeline.drawFrom(Sources.<Long, String>map(LINES)) .map(e -> e.getValue().toLowerCase()) .flatMap(t -> traverseArray(delimiter.split(t))) .filter(word -> !word.isEmpty()) .groupingKey(wholeItem()) .aggregate(counting()) .drainTo(Sinks.map(COUNTS)); return pipeline;
- 25. Example 1: Word Count
- 26. Example 1: Word Count
- 27. Example 1: Word Count Hazelcast Jet Pattern delimiter = Pattern.compile("W+"); Pipeline pipeline = Pipeline.create(); pipeline.drawFrom(Sources.<Long, String>map(LINES)) .map(e -> e.getValue().toLowerCase()) .flatMap(t -> traverseArray(delimiter.split(t))) .filter(word -> !word.isEmpty()) .groupingKey(wholeItem()) .aggregate(counting()) .drainTo(Sinks.map(COUNTS)); return pipeline;
- 28. Example 1: Word Count Demo: https://github.com/hazelcast/hazelcast-jet-code-samples
- 29. Part 2: Jet Under the Hood
- 30. How does it work?
- 31. How does it work?
- 32. How does it work?
- 33. How does it work? Under the Hood: ● Generate DAG representation from Pipeline ● Serialize DAG ● Send DAG to every Node ● Deserialize DAG ● Executes DAG on each Node
- 35. Infinite Streams
- 36. Infinite Streams Examples: ● Currency Exchange Rates ● Tweets from Twitter ● Events in some Event-Based system ● ...
- 38. Example 2: Twitter Cryptocurrency Analysis Problem: Present in real-time the sentiments about cryptocurrencies Input: Tweets are streamed from Twitter and categorized by coin type (BTC, ETC, XRP, etc) Output: Tweets sentiments (last 30 sec, last minute, last 5 minutes)
- 39. Example 2: Twitter Cryptocurrency Analysis Demo: https://jet.hazelcast.org/demos/
- 40. Part 3: Jet in the Cloud
- 41. Jet in the Cloud: discovery plugins
- 42. Jet in the Cloud: discovery plugins
- 43. Jet in the Cloud: discovery plugins
- 44. Jet in the Cloud: discovery plugins
- 45. Jet in the Cloud: discovery plugins
- 46. Jet in the Cloud: discovery plugins
- 47. Jet in the Cloud: deploying on k8
- 48. Jet in the Cloud: deploying on k8 $ helm install stable/hazelcast-jet
- 49. Jet in the Cloud: deploying on k8 $ kubectl scale <name> --replicas=6
- 50. Example 3: Stock Trade Aggregator Problem: Present in real-time the aggregated trade price of stocks Input: Stock trades with name and price Output: Sum of prices per stock name
- 51. Example 3: Stock Trade Aggregator Demo: https://github.com/hazelcast/hazelcast-jet-code-samples/t ree/master/integration/kubernetes
- 52. Part 4: Jet Features & Use Cases
- 53. Jet Features Categories of Features ● Easy to Use ● Performance
- 56. Jet Features: other features
- 57. Why would I need it? ● Big Data Projects
- 58. Why would I need it? ● Big Data Projects ● Speed up Everything
- 59. Why would I need it? ● Big Data Projects ● Speed up Everything
- 60. Example 4: Web Crawler Problem: Parse all blog posts from the webpage Input: URL of Blog Trips Output: All the content from the Blog
- 61. Example 4: Web Crawler Demo: https://github.com/leszko/geodump
- 62. Thank You!