IMCSummit 2015 - Day 2 Developer Track - Implementing a Highly Scalable In-Memory Stock Prediction System with Apache Geode, R and Spring XD
6 likes1,618 views
The document discusses a stock prediction system utilizing open-source software, focusing on machine learning techniques like neural networks and real-time data processing. It highlights challenges in traditional models and emphasizes the need for real-time, stream-based analytics. The architecture proposed includes the integration of various data sources with a framework for scalable and efficient processing using tools like Spring XD and GemFire.
![Info
Analysis
Filter
[ json ]
Neural Network
How can it be addressed?](https://image.slidesharecdn.com/imcs20152devimplementingahighlyscalablein-memorystockprediction-150716213744-lva1-app6892/85/IMCSummit-2015-Day-2-Developer-Track-Implementing-a-Highly-Scalable-In-Memory-Stock-Prediction-System-with-Apache-Geode-R-and-Spring-XD-11-320.jpg)
IMCSummit 2015 - Day 2 Developer Track - Implementing a Highly Scalable In-Memory Stock Prediction System with Apache Geode, R and Spring XD
- 1. © Copyright 2014 Pivotal. All rights reserved.© Copyright 2014 Pivotal. All rights reserved. A Stock Prediction System using open-source software Fred Melo [email protected] @fredmelo_br 1 William Markito [email protected] @william_markito
- 3. © Copyright 2014 Pivotal. All rights reserved. It's all about DATA Data Sources Look for patterns Prediction
- 4. © Copyright 2014 Pivotal. All rights reserved.
- 5. © Copyright 2014 Pivotal. All rights reserved. 5 Machine Learning is the answer Neural Networks Clustering Genetic Algorithms
- 6. © Copyright 2014 Pivotal. All rights reserved. Train with historical dataset Apply model to the new input Applying Machine Learning
- 7. Hard to add new data sources Why? Hard to scale Why so hard? Hard to make it real-time
- 8. Traditional models are reactive and static HDFS Data Lake Store Analytics Hard to change Labor intensive Inefficient No real-time information ETL based Data-source specific
- 9. Stream-based, real-time closed-loop analytics are needed HDFSData Lake Expert System / Machine Learning In-Memory Real- Time Data Continuous Learning Continuous Improvement Continuous Adapting Data Stream Pipeline Multiple Data Sources Real-Time Processing Store Everything
- 10. Info Analysis Look at past trends (for similar input) Evaluate current input Score / Predict Neural Network How can it be addressed?
- 11. Info Analysis Filter [ json ] Neural Network How can it be addressed?
- 12. Info Analysis Filter Enrich Neural Network How can it be addressed?
- 13. Info Analysis Neural Network Filter Enrich Transform How can it be addressed?
- 14. Info Analysis Filter Enrich Transform Neural Network How can it be addressed?
- 15. Info Analysis Filter Enrich Transform Transform Neural Network How can it be addressed?
- 16. Neural Network In-Memory Data Grid Real-time scoring How can it be addressed? Train
- 17. Neural Network In-Memory Data Grid Front-end Update Push How can it be addressed?
- 18. Ingest Transform Sink SpringXD Store / Analyze Fast Data Distributed Computing Predict / Machine Learning Other Sources and Destinations JMS Streaming real-time analytics architecture
- 19. Transform Sink SpringXD Extensible Open-Source Fault-Tolerant Horizontally Scalable Cloud-Native HTTP Machine Learning Fast Data Filter Predict Sink HTTP Split Dashboard Push Demo Architecture
- 20. SpringXD INGEST / SINK PROCESS ANALYZE • Little or no coding required • Dozens of built-in connectors • Seamless integration with Kafka, Sqoop • Create new connectors easily using Spring • Call Spark, Reactor or RxJava • Built-in configurable filtering, splitting and transformation • Out-of-box configurable jobs for batch processing • Import and invoke PMML jobs easily • Call Python, R, Madlib and other tools • Built-in configurable counters and gauges Data Stream Pipelining
- 21. SpringXD XD NodesXD NodesXD NodesXD Nodes Ingest SpringXD Split Filter Transform Sink XD admin XD Nodes Ingest Split Filter Transform Sink Stream Deployment Messaging Scale-Out and HA Architecture
- 22. Transform Sink SpringXD Extensible Open-Source Fault-Tolerant Horizontally Scalable Cloud-Native HTTP Machine Learning Fast Data Filter Predict Sink HTTP Split Dashboard Push Demo Architecture
- 24. Partitioned Regions GemFire Server1 GemFire Server2 Primary'to'redundant'replica1on' Primary 0 2 4 6 Redundant 1 3 5 7 Region A Region B Region A Region B Primary 1 3 5 7 Redundant 0 2 4 6
- 25. Event handling GemFire'Server' Region'A' ' '''''''''' subscrip4on Region'A' pool6name=ServerPool' ' ''''''''' X GemFire'Client'1' pool'"ServerPool"' (with'or'without'' subscrip4ons'enabled) Region'A' pool6name=ServerPool' ' ''''''''' X GemFire'Client'2' pool'"ServerPool"' (with'subscrip4ons'enabled,' 'interest'register'in'X,'receiveValues=true) X Distributed'System Update'/'Create 1 2 3 3 4 X The pool propagates the event to the cache server, where the region is updated. The server distributes the event to its peers and also places it into the subscription queue for Client 2.
- 26. Transform Sink SpringXD Extensible Open-Source Fault-Tolerant Horizontally Scalable Cloud-Native HTTP Machine Learning Fast Data Filter Predict Sink HTTP Split Dashboard Push Demo Architecture
- 27. Neural Networks
- 28. Neural Networks
- 29. medium avg (x+1) relative strength (x) medium avg (x) price(x) Neural Network
- 30. Neural Network
- 31. Transform Sink SpringXD Extensible Open-Source Fault-Tolerant Horizontally Scalable Cloud-Native HTTP Machine Learning Fast Data Filter Predict Sink HTTP Split Dashboard Push Demo Architecture
- 33. Demo Time
- 37. A NEW PLATFORM FOR A NEW ERA