Mantis qcon nyc_2015
- 1. Mantis in Action Neeraj Joshi and Justin Becker 6/12/2015
- 2. Managing a complex operational environment is hard
- 6. Developing an understanding of what is going on Knowing what works
- 7. Developing an understanding of what is going on Identify what doesn’t work
- 8. Developing an understanding of what is going on Determining impact when doesn’t work
- 9. Developing a deep understanding is hard, due to complexity • Hundreds of software services • Processing billions of requests • For millions of users • Operating in multiple data centers • Across the globe
- 10. Help us (operators) make sense out of complexity, need tools specifically, we need… Insight tools …to help comprehend what is going on in our operational environments
- 11. Make the case a relationship exists, between complexity and comprehension
- 12. So, in order to manage complex environments, need to rethink insights, shift the curve
- 13. Identified three insight ‘patterns’ to help shift the curve • Long tail analysis • Real time tracking and trending • Ad hoc investigation
- 14. Grouped three patterns into a new effort
- 16. Goal is to help us manage (comprehend) our environments given an increase in complexity
- 17. Mention specific insights tools to help shift curve • Realtime Data Explorer • Realtime Search • Realtime Application Monitoring
- 18. Mention other generic insights jobs to help shift curve • Short term historical anomaly detection • Threshold-based anomaly detection • Realtime metrics generation
- 19. Overview of scalable insights in action • 65-75 total jobs running in 3 regions • Global access to data • Processing 4.7 million events per second at peak • 20 specific data sources and generic adapters • Ability to startup jobs in ~5 seconds
- 20. Demo
- 25. Mantis, a reactive stream processing system
- 27. Stream Sequence of Events time 1 2 3 4
- 28. Higher Order Functions Transformations applied to a Stream to create a new stream time 1 4 9 16 map (x=>sqrt(x)) time 1 2 3 4
- 29. S O U R C E Stream IN S I N K Result OUT Stage 1 Stage 2 Stage n A sequence of functions applied to a stream map window reduce merge Mantis Job
- 30. Mantis Job S O U R C E Stream IN S I N K Result OUT Stage 1 Stage 2 Stage n
- 33. Have SLAs Min/Max Instances of the job Min/Max runtime Perpetual or Transient
- 34. Can be chained Device Logs Source Job TopN Job Server Logs Source Job Anomaly Detector Job Metrics Aggregator Job Alert Service
- 35. That is fine but...
- 36. How does Mantis meet the Scalable Insights challenge?
- 37. • Cost (Utilization) Sensitive • Optimize for low latency • High Throughput • Resilient Key Requirements
- 38. Minimizing Costs
- 39. Elastic Clusters
- 40. Elastic Jobs
- 41. Job Autoscaling Scaling Config CPU Strategy Network Strategy
- 42. Filtering at Source Source Job Consumer Job Data Producers
- 43. Low latency - High Throughput
- 44. To block or not to block?
- 45. RxNetty (non-blocking) vs Tomcat (blocking) by Brendan Gregg @brendangregg https://docs.google.com/presentation/d/18i-d72m7tD4wKlzm-1PCR8g62l66_9Btbg5-fuFRqf0/edit#slide=id.g761289dab_0_77
- 46. CPU consumption • RxNetty consumes less CPU / request • Reduced thread migration • Lower object allocation rate CPU consumed reduces as load increases
- 47. Lower latency • RxNetty has lower latency under high load • fewer lock contentions • fewer thread migrations Latency knee for Tomcat ~ 400 Latency knee for Netty ~700
- 48. Async Processing Non-blocking I/O Async Processing
- 50. Server Resilience • Servers crashes inevitable • Server health constantly monitored with heartbeats • Crashed servers replaced • Lost jobs relaunched
- 51. Network Resilience • Long lived connections can fail • Connection topology is constantly monitored and corrected
- 52. Backpressure
- 54. Hot Source
- 55. Reactive push-pull (Cold Source) f1 f2 f3 f4 Cold Source 102410241024 1024 100 100 100 100
- 56. Push mode f1 f2 f3 f4 Cold Source MaxMaxMax Max 100 100 100 100
- 57. Pull mode f1 f2 f3 f4 Cold Source 111 1 1 1 1 1
- 58. Backpressure Strategies (Hot Source) f2 f3 f4 Hot Source 101010 100 10 10 10 Strategy Function 90 Drop Buffer Scale-up
- 59. Questions