Open Data Hub - Matteo Roggia - Animeshon - Real-time Open Data Hub.pdf
0 likes48 views
The document outlines the architecture and functionalities of the Open Data Hub, a modern, distributed system employing microservices for efficient data processing and communication. It highlights the system's event-driven design, fault tolerance, and adaptability to meet evolving requirements, while emphasizing its open-source nature and ease of integration. The Open Data Hub promotes innovation through standardized protocols, modularity, and the ability for users to experiment with data using no-code tools.
Open Data Hub - Matteo Roggia - Animeshon - Real-time Open Data Hub.pdf
- 1. Open Data Hub ARCHITECTURE DESIGN BY ANIMESHON
- 2. 01 – Open Data Hub 02 – Technologies 03 – A new way to communicate 04 – Reactivity 05 – Plug & Research 00 – Introduction AGEND A 06 – Conclusion
- 4. Like to a post send notification to users weather forecast model improve future forecasts notify consumers recommendation engine recommend ads update feed update profiling information Resolve Captcha IoT sensor sending pressure feedback to AI models protect from unauthorized actors collect data for models training
- 5. Open Data hub A modern system which already serves thousands of citizens while processing data from dozens of providers Open Data Hub v2 A system designed to meet future requirements while reaching more people with more and faster information State of the art technologies. International standards enforcement
- 7. A distributed, event-driven, zero-downtime system Open Data Hub:
- 8. distributed microservices Open Data Hub is a cluster of countless microservices single responsibility Each service has only one responsibility which can be carried out in the most efficient way
- 9. event-driven async communication The subprocess chain communicates asynchronously, making each component independent from the others service self-healing Events are thrown in the system, and the consumer can read them when they are ready. If something goes down, events are preserved until it comes back online agnostic communication Services do not know who will be the next process to take control, they just inform the system about their own results
- 10. zero-downtime no data loss Once an information enters the system, it will be preserved and backed up to ensure data integrity and data durability update rollout Service orchestration allows the team to rollout updates with different strategies zero or minimal downtime fault tolerant Even if multiple services are offline, the system can still accept data and serve results
- 11. 02 TECHNOLOGIES
- 12. no vendor lock-in All technologies are provider agnostic and can be migrated to any private or public cloud provider open source All technologies are open source easy to contribute Open Data Hub’s public repository allows anyone to start a local cluster of the core architecture and start contributing
- 13. anyone can locally instantiate an Open Data Hub cluster in just 20 minutes and start contributing to the project https://github.com/noi-techpark/odh-infrastructure-v2
- 14. 02 A NEW WAY TO COMMUNICATE
- 15. Open Data Hub uses a standard communication protocol to both serve and process incoming data in alignment with internet 2.0 common practices standard communication protocol Speaking a common language makes it more appealing for third parties to integrate with Open Data Hub easy to integrate A more structured and standardized communication protocol allows for a reduction of custom solutions and more precise specifications open specifications
- 16. RestApi Open Data Hub enforces communication standards with providers. MQTT Websocket
- 17. 04 REACTIVITY
- 18. easy to maintain Changing the order of the blocks, or substitute some of them will not affect the rest of the system The whole architecture is composed by small lego-blocks modular design
- 19. All lego-blocks can be rearranged in whatever form to satisfy new requirements adaptability The system is designed to react to future needs and evolve evolution by design
- 21. Open Data Hub allows for isolated environments to utilize historical raw data Thanks to no-code tools researches could create data processing pipeline without IT knowledge Allowing institutions to experiment with both processed and raw data enables an exponential grow of the system itself sandbox semplicity grow
- 22. 06 CONCLUSION
- 23. Open Data Hub promotes open source by adopting only open technologies while publishing it’s core infrastructure open State-of-the-art technologies and modern paradigms ensure an easy maintenance and integrability innovative Designed to meet any present and future requirement scalable
- 24. Thank you