Ted Dunning-Faster and Furiouser- Flink Drift
2 likes•840 views
The document discusses MapR Technologies' converged platform for big and fast data, which integrates files, streams, and tables with open-source technology to enhance performance and scalability. It highlights the importance of a streaming-first architecture, demonstrating the improvement of data processing speed through various tuning methods and real-time application scenarios. The presentation concludes with insights into the future potential for performance improvements and the evolving landscape of data storage solutions.
Ted Dunning-Faster and Furiouser- Flink Drift
- 1. © 2014 MapR Technologies 1© 2014 MapR Technologies
- 2. © 2014 MapR Technologies 2 Me, Us • Ted Dunning, MapR Chief Application Architect, Apache Member – Committer PMC member Zookeeper, Drill, others – Mentor for Flink, Beam (nee Dataflow), Drill, Storm, Zeppelin – VP Incubator – Bought the beer at the first HUG • MapR – Produces first converged platform for big and fast data – Includes data platform (files, streams, tables) + open source – Adds major technology for performance, HA, industry standard API’s • Contact @ted_dunning, [email protected], [email protected]
- 3. © 2014 MapR Technologies 3 New book on Apache Flink Download free pdf courtesy of MapR Technologies mapr.com/flink-book
- 4. © 2014 MapR Technologies 4 Agenda • Why streaming first architecture • What does fast mean? • How do I make something fast? • Minor pause for reality check • First steps … heavy bottlenecks • Real results • Deeper insights
- 5. © 2014 MapR Technologies 5 Is this really a revolutionary moment?
- 6. © 2014 MapR Technologies 6 Scenario: Profile Database
- 7. © 2014 MapR Technologies 7 The task ? POS 1 location, t, card # yes/no? POS 2 location, t, card # yes/no?
- 8. © 2014 MapR Technologies 8 Traditional Solution POS 1..n Fraud detector Last card use
- 9. © 2014 MapR Technologies 9 What Happens Next? POS 1..n Fraud detector Last card use POS 1..n Fraud detector POS 1..n Fraud detector
- 10. © 2014 MapR Technologies 10 What Happens Next? POS 1..n Fraud detector Last card use POS 1..n Fraud detector POS 1..n Fraud detector
- 11. © 2014 MapR Technologies 11 How to Get Service Isolation POS 1..n Fraud detector Last card use Updater card activity
- 12. © 2014 MapR Technologies 12 New Uses of Data POS 1..n Fraud detector Last card use Updater Card location history Other card activity
- 13. © 2014 MapR Technologies 13 Scaling Through Isolation POS 1..n Last card use Updater POS 1..n Last card use Updater card activity Fraud detector Fraud detector
- 14. © 2014 MapR Technologies 14 For this to work (socially), streaming has to be faster than almost any requirement
- 15. © 2014 MapR Technologies 15 So how do we make something go really fast?
- 16. © 2014 MapR Technologies 16 Make some data Process itmove it
- 17. © 2014 MapR Technologies 17 Make some data Process itmove it World domination move it
- 18. © 2014 MapR Technologies 18 Well, perhaps not quite so simple?
- 19. © 2014 MapR Technologies 19 Interactive recommendation query db Off-line analysis Real-time event source Recent history Item linkage Search Recommendations Cooccurrence analysis Long- term history queue queue Recommendations
- 20. © 2014 MapR Technologies 20 mySQL mySQL files Web-site Auth service Upload service Image extractor Transcoder User profiles Search User action logging Recommendation analysis mySQL mySQL mySQL Oracle Solr Elastic User Generated Content
- 21. © 2014 MapR Technologies 21 Yahoo Streaming Benchmark Ad server Filter Group by campaign impressions Campaign info Count impressions Results Project Augment Window by event time
- 22. © 2014 MapR Technologies 22 Ad server Filter Group by campaign impressions Campaign info Count impressions Results Project Augment Window by event time Client lock Partitions Threads/machineShuffle
- 23. © 2014 MapR Technologies 23 Ad server Filter Group by campaign impressions Campaign info Count impressions Results Project Augment Window by event time Client lock Partitions Threads/machineShuffle Threads/machineShuffle
- 24. © 2014 MapR Technologies 24 Ad server Filter Group by campaign impressions Campaign info Count impressions Results Project Augment Window by event time Ad server impressions Filter Project Augment Group by campaign Count impressions Client lock Partitions Threads/machineShuffle Threads/machineShuffle
- 25. © 2014 MapR Technologies 25 What we do at MapR
- 26. © 2014 MapR Technologies 26 Evolution of Data Storage Functionality Compatibility Scalability Linux POSIX Over decades of progress, Unix-based systems have set the standard for compatibility and functionality
- 27. © 2014 MapR Technologies 27 Functionality Compatibility Scalability Linux POSIX Hadoop Hadoop achieves much higher scalability by trading away essentially all of this compatibility Evolution of Data Storage
- 28. © 2014 MapR Technologies 28 Evolution of Data Storage Functionality Compatibility Scalability Linux POSIX Hadoop MapR enhanced Apache Hadoop by restoring the compatibility while increasing scalability and performance Functionality Compatibility Scalability POSIX
- 29. © 2014 MapR Technologies 29 Functionality Compatibility Scalability Linux POSIX Hadoop Evolution of Data Storage Adding converged tables and streams enhances the functionality of the base file system
- 30. © 2014 MapR Technologies 30 http://bit.ly/fastest-big-data
- 31. © 2014 MapR Technologies 31 Key Ideas • Convergence of files, tables, streams into single platform – All forms of persistence share common implementation base • Very high abstraction from hardware … no need to provision clusters for tables and files – Common disaster recovery, security, availability models for files, directories, tables and streams • Very high performance levels
- 32. © 2014 MapR Technologies 32 Key Issues • MapR itself is heavily threaded internally (as many as 50k threads/core) • MapR client can have multiple internal threads • Ordering boundaries require serialization, locks or memory contention – At client level and also within single stream/topic/partition • Replication, splitting, data location completely automated by default, explicit control available • MapR Streams and Flink are in same cluster, but some shuffles still required
- 33. © 2014 MapR Technologies 33 Initial Configuration • 10 nodes in cluster • 1 Flink task manager / node • 72 partitions in impressions stream • Each task manager spawns 72 generator threads Ad server impressions Ad server impressions 10x72 threads 72 partitions • At full speed, partition insert points wander around cluster to avoid hot-spotting • MapR client connection shared by all threads in task manager. Having more client connections could help
- 34. © 2014 MapR Technologies 34 Tuning #1 • Large number of threads and single client connection per node caused massive contention at serialization point inside client • Switched to 3 Flink task managers per node • 2 task managers each run 1 producer thread – More data pushed by 1 thread than previously sent by 72
- 35. © 2014 MapR Technologies 35 Tuning #2 • Effective cluster-wide parallelism limited by 72 partitions in stream • Increasing to 300 partitions substantially improved performance
- 36. © 2014 MapR Technologies 36 The consumer • Initial tuning had 72 consumer threads per node • Final tuning used single consumer thread per Flink task manager Filter Campaign info Project Augment Filter Project Augment
- 37. © 2014 MapR Technologies 37 The Shuffle / Group-by • Shuffles were also run by the single consumer task manager • Even with shuffle, consumer processes balanced producer processes Group by campaign Count impressions Results Window by event time Group by campaign Count impressions
- 38. © 2014 MapR Technologies 38 Tuning #3 • In separate experiments, number of campaigns was increased to 1e6 from original 100 • This caused bottle neck to shift massively to data export step • Serving results directly from Flink memory avoids this step
- 39. © 2014 MapR Technologies 39 Final Comparisons Flink on MapR no tuning Transactions / second (millions) 0 5 10 15 Flink on MapR tuned Final result for tuning was 250% improvement No serious optimization was required, however
- 40. © 2014 MapR Technologies 40 The Moral • Default of 10 partitions per topic is fine for large-scale multi- tenancy, but special purpose applications may need tuning to higher levels (we ended up with 30 partitions per node) • Asynchronous client gives effective threading with small number of producer threads, large number of producer threads was counter-productive • Net speedup of 250% with tuning, so far • Gut feel is that there is ~4x more performance still to come
- 41. © 2014 MapR Technologies 41 Me, Us • Ted Dunning, MapR Chief Application Architect, Apache Member – Committer PMC member Zookeeper, Drill, others – Mentor for Flink, Beam (nee Dataflow), Drill, Storm, Zeppelin – VP Incubator – Bought the beer at the first HUG • MapR (www.mapr.com) – Produces first converged platform for big and fast data – Includes data platform (files, streams, tables) + open source – Adds major technology for performance, HA, industry standard API’s • Contact @ted_dunning, [email protected], [email protected]
- 42. © 2014 MapR Technologies 42 New book on Apache Flink Download free pdf courtesy of MapR Technologies mapr.com/flink-book
- 43. © 2014 MapR Technologies 43 Streaming Architecture by Ted Dunning and Ellen Friedman © 2016 (published by O’Reilly) Free signed hard copies at MapR booth at Flink Forward http://bit.ly/mapr-ebook-streams
- 44. © 2014 MapR Technologies 44 Short Books by Ted Dunning & Ellen Friedman • Published by O’Reilly in 2014 - 2016 • For sale from Amazon or O’Reilly • Free e-books currently available courtesy of MapR Download pdfs: mapr.com/ebooks-pdf
- 45. © 2014 MapR Technologies 45 Thank You!
- 46. © 2014 MapR Technologies 46 Q&A @mapr maprtech [email protected] Engage with us! MapR maprtech mapr-technologies