Lessons From HPE: From Batch To Streaming For 20 Billion Sensors With Lightbend Platform
Download as PPTX, PDF2 likes1,215 views
The document outlines the transition of 3PAR storage arrays from batch processing to a streaming architecture to enhance data handling capabilities, addressing diverse data rates from numerous sensors. It emphasizes the benefits of a reactive architecture that supports scalability, resilience, and fast recovery from failures while leveraging technologies like Apache Kafka, Cassandra, and Akka. The new system significantly improves customer satisfaction and reduces costs by enabling near real-time updates and automated support capabilities.
Lessons From HPE: From Batch To Streaming For 20 Billion Sensors With Lightbend Platform
- 1. 3PAR Streaming Journey Chris McDermott January 2020
- 2. Agenda 1. The 3PAR use case 2. Batch to Streaming transition 3. Reactive architecture, micro-services and event sourcing 2
- 3. The 3PAR Use case • 50K 3PAR Storage Arrays (SAs) • Different data types have different arrival rates • 500 GB/day on average • Need to process 10x bursts • 10K sensors per SA • ~10 sources of enhancement data 3
- 4. The 3PAR Use case (continued) • Joins! • Analytics • Statistical aggregations • Prediction • Projections • Automated case management • Legacy Integrations • Create event sources from non-eventing services using Reactive facades 4
- 5. Batch Vs Streaming Day 1 Processing Data Day 2 Processing Data Day 3 Processing Data Day 4 Day N Day N Processing Data Day 1 Input Data Day 3 Input Data Day 2 Input Data Day 4 Input Data Time Lag T 1 Processing Data T 2 Processing Data T 3 Processing Data T 4 Time N Time N Processing Data T 1 Input Data T 3 Input Data T 2 Input Data T 4 Input Data No Time Lag Batch Streaming
- 6. Batch Processing • Serial processing of all data on a regular cadence * Push = project data to new Elasticsearch index • As the amount of history and systems increase, each stage of the pipeline takes longer to run • Failures take a long time to recover from (Push failure at the 20-hour mark…) • Large quanta problems means repeating failed changes takes a long time • Based on Spark, so monitoring is sub-par 6 Gather (1 hour) Process (4 hours) Push* (30 hours)
- 7. Streaming Processing • Parallel processing of (relatively) small chunks of data (per SA) as soon as the data is received • Data lag, or data freshness, is always consistent at less than 5 minutes. New data is always processed as soon as it is available. • Failure recovery is extremely fast oWhen running at 10x line rate, full recovery is roughly 10% of outage time (2-day outage is recovered in ~5 hours) oCan dynamically apply more resources to increase processing performance • Based on Lagom/Akka, which provides built-in metrics and reporting framework 7
- 8. Reactive Architecture and Technologies
- 9. Reactive Architecture “Systems built as Reactive Systems are more flexible, loosely-coupled and scalable. This makes them easier to develop and amenable to change. They are significantly more tolerant of failure and when a failure does occur they meet it with elegance rather than disaster. Reactive Systems are highly responsive, giving users effective interactive feedback.” 9 Reactive Systems Are: • Responsive: The system responds in a timely manner if at all possible. • Resilient: The system stays responsive in the face of failure. This applies not only to highly-available, mission-critical systems — any system that is not resilient will be unresponsive after a failure. • Elastic: The system stays responsive under varying workload. Reactive Systems can react to changes in the input rate by increasing or decreasing the resources allocated to service these inputs. • Message Driven: Reactive Systems rely on asynchronous message-passing to establish a boundary between components that ensures loose coupling, isolation and location transparency. Summarized from the Reactive Manifesto
- 10. 3PAR Streaming – Technology Stack 10 Apache Kafka – durable, elastic, fault-tolerant, log based, message bus Apache Cassandra – durable, elastic, fault-tolerant noSQL database Elasticsearch – durable, elastic, fault-tolerant document-store optimized for search Apache NiFi – dataflow automation with graphical programming interface Akka - a toolkit for building highly concurrent, distributed, and resilient message- driven applications Play – web based (REST) application framework based on Akka
- 11. 3PAR Streaming – Technology Stack 11 Lightbend Commercial Components • Split Brain Resolver • Telemetry • Thread Starvation Detector
- 12. Telemetry
- 13. 3PAR Simplified Streaming Component Architecture Ingest Support Tickets Entitlement ML Application NiFi Support Lagom Entitlement Lagom ES Projector * * … … Elasticsearch StoreServ API StoreServ Akka InfoSight UI HPE
- 14. Akka StoreServ Akka • Device shadow model • Stores raw data in Cassandra (data lake) • Stores Actor State in Cassandra • Actors cache most recent data in memory for very low latency • Actors are rehydrated from State in Cassandra • Actors are not passivated • Scale out by adding more instances when running out heap
- 15. Akka vs Lagom Various stateful micro services written using Lagom • Lagom makes sense for event driven micro services • If you can store the entire event history and rebuild the read-side from it in a reasonable amount of time (event sourcing) • Most use cases fall into this category. • Plain Akka makes more sense if you can’t afford to save the entire event history or rebuilding the read- side from the event history is too expensive. Or you simply don’t need the entire event history to rebuild the read-side. • Persisted the entire read-side (Kafka) • Still CQRS (but not event-sourced.)
- 16. ES Projector • Akka Streams application • Reads full data model • Creates “role” based projections of data into Elasticsearch StoreServ API • Uses Play Framework • Basically provides a thin wrapper over Elasticsearch queries • Modifies client queries to enforce access control (both tenancy and role restrictions)
- 17. Results
- 18. What was gained? • Responsive: InfoSight is updated in near real-time • Reduced lag gives customers greater confidence • Allows automated support (problems can be remediated sooner: outages are prevented) • Resilient: InfoSight is more reliable • Microservice isolation means the system degrades instead of totally fails. • Elastic: Containerization and Scale-out technologies • The system can easily be scaled to account for growth and new processing • Message Driven: Well defined boundaries and client managed message consumption • Isolated components are more easily understood. • New components can be added without changing the underlying architecture.
- 19. Bottom Line • Customer satisfaction is increased • HPE costs are decreased
- 20. Futures
- 21. Data Platform Goals • Allow the on-boarding of a disparate set product lines quickly and efficiently • Data-lake to share data across internal organizations • Support exploratory analytics – Data democracy • Access Control and Multitenancy (RBAC) • Uniform Data Access API • Support for ML workflows
- 22. Data Platform • Scala • Kubernetes • S3 • Delta-Lake • Spark • Akka • Kafka • Cassandra/ElasticSearch/PostgresSQL,
- 23. Are you ready?
- 24. HPE is Hiring Click on the links below to see job descriptions – note: the URLs are subject to change. For the latest information, visit https://careers.hpe.com https://careers.hpe.com/job/Hewlett-Packard-Enterprise-Andover-Massachusetts/91589424 https://careers.hpe.com/job/Hewlett-Packard-Enterprise-Andover-Massachusetts/91589425 https://careers.hpe.com/job/Hewlett-Packard-Enterprise-Andover-Massachusetts/91589426 https://careers.hpe.com/job/Hewlett-Packard-Enterprise-Andover-Massachusetts/91589427 https://careers.hpe.com/job/Hewlett-Packard-Enterprise-Andover-Massachusetts/91589423 You may apply through the career site or send resumes directly to [email protected]
Editor's Notes
- #4: 3PARs are high end storage arrays
- #7: Batch failures are a “vey large quanta” problems. Failure in a batch stage requires reprocessing of that very large quanta. Push needs to copy the Vertica index, needs to pull in all historical data, plus all newly processed data into a new index. Spark monitoring is poor because much of it sits behind Yarn. We also have no ssh level access to the HDP nodes.
- #8: Streaming failures are a ”small quanta” problem. Recovery requires only reprocessing of that small quanta. Streaming push - pushes new data as it becomes available. There is no “dump truck” of data that needs to be indexed which leads to a more available and stable Elasticsearch cluster. No index rolling and copying data between an old index and a new index.
- #10: Reactive Architecture arose from the rejection of a model where remote communication was trying to be disguised as local: e.g. rRMI, CORBA, DCE, etc. Reactive Architecture fully accepts the realities of distributed systems by never treating anything as local. There are no synchronous messages. Everything is location transparent and assumed to be remote. Failures to occur and the system still must remain as available and as functional as possible.
- #11: Akka supports multiple processing paradigms including streaming. Lagom is an opinionated API framework built on top of Akka.
- #12: Akka supports multiple processing paradigms including streaming. Lagom is an opinionated API framework built on top of Akka.
- #14: Kafka provides centralized message bus comprised of many channels (topics) Many more data sources adapted to Kakfa by Akka Streaming/Lagom: iBase, BL/WL, CFST (crash file search tool), DSPN Listener We are using Cassandra for all statefull Akka based micro-services, but other than storeserv lagom, it is only used in those for state persistence. For storeserv_lagom it provides “random access” to the raw files that also live in Kafka. Also stores perform-deltas which are too ”large” for elasticserach.
- #20: Near real-time analytics allows us to fix customer problems before they result in outages
- #22: Near real-time analytics allows us to fix customer problems before they result in outages
- #23: Near real-time analytics allows us to fix customer problems before they result in outages