Intelligently Monitor And Rapidly Troubleshoot Streaming Fast Data Applications With OpsClarity
2 likes1,472 views
The document outlines the agenda for a discussion on fast data and streaming applications, emphasizing the challenges and requirements for effective monitoring solutions. Key topics include monitoring various health aspects of data applications, the inadequacy of traditional monitoring tools, and the necessity for deep telemetry and intelligent anomaly detection. It concludes with an invitation for a demo to explore end-to-end monitoring solutions by Lightbend.
More Related Content
Similar to Intelligently Monitor And Rapidly Troubleshoot Streaming Fast Data Applications With OpsClarity (20)
Intelligently Monitor And Rapidly Troubleshoot Streaming Fast Data Applications With OpsClarity
- 2. Agenda 1. Fast Data & Streaming Applications 2. The Challenges of Monitoring Fast Data Applications 3. What To Look For In a Fast Data Application Monitoring Solution 4. Intelligent End-To-End Monitoring from Lightbend 5. Live Demo 6. Questions
- 5. reactivemanifesto.org Reactive Underpinnings: Fast Data and streaming applications often incorporate, or are based on, Reactive principles
- 6. • Rapidly Evolving Ecosystem • Understanding the Data Pipeline • Dynamic Architectures • Intricately Interconnected • Distributed And Clustered The Challenges of Monitoring Fast Data Applications
- 7. Apache Spark, As An Illustrative Example The Challenges of Monitoring Fast Data Applications Concern Questions To Ask Data Health (for a given application) • Throughput: is data processing occurring at the expected rate? • Latency: is data processing occurring within the expected timeframe? • Error/quality: are there problems with the data being produced? • Input data: are input data streams flowing into Spark behaving normally? For instance, what are the throughput rates for Kafka topics feeding into the Spark job? Dependency Health • Are the systems feeding input into the spark job (such as Kafka) healthy? • Are the systems that the application is dependent on, such as Memcache or other API endpoints, healthy? Service Health • Is the Spark master operating normally? If not, engineering will be unable to re-balance workloads or restart jobs. Application Health • Are the application KPIs within normal operating parameters? Topology Health • Are there resources assigned to the given Spark topology? • • Are the Spark tasks and executors well-distributed amongst the Spark cluster? • • Are the performance counters (emitted, failed, latency, etc.) for the given Spark topology normal? Node System Health • Are the key system metrics (load, CPU, memory, net-i/o, disk-i/o, disk free) operating normally?
- 8. Modern monitoring for modern applications
- 9. Why traditional monitoring tools won’t help you • Built to monitor monolithic applications • Can only be used to extract metrics and trace information based on a synchronous flow • Not built for asynchronous flows (i.e. in Fast Data and streaming applications) • Cannot easily handle streaming systems running on distributed clusters
- 10. • Automatic Telemetry • Visual Maps • Intelligent, Rapid Troubleshooting What users need to effectively monitor Fast Data and streaming applications
- 11. • Deep Telemetry • Domain Expertise • Intelligent Anomaly Detection • Fine-Grained Visibility, with Drill- Down Capabilities Data-Science Driven Anomaly Detection
- 12. • Automated Topology Discovery • Automatic Metric Collection • Real-Time Topology Visualization Automated Discovery, Configuration & Topology Visualization
- 13. • Single Pane of Glass Visibility • Rapid Root Cause Analysis • Reduced Mean-Time- To-Repair (MTTR) Intelligent, Rapid Troubleshooting
- 14. • Dramatically reduce the time and cost to identify and remediate issues across application life-cycle. • Create happier, more satisfied customers – and lower churn • Lower HW/infrastructure costs and reduce concerns about chargebacks & SLA penalties • Deliver rapid time to value because everything you need for monitoring is packaged into an easy-to-use solution Benefits for your business
- 15. On to the demo…
- 17. If you’re serious about having end-to-end monitoring for your Fast Data and streaming applications, let’s chat! SET UP A 20-MIN DEMO