![©
Fraunhofer
12
ISCRAM 2013, Baden-Baden, Germany
4) Scalable and high performance storage of large distributed
datasets
Design decision
Working datasets
Hybrid databases [OWLIM for
metadata, MySQL for data] –
semantically rich and efficient
queries, clients must understand two
protocols (SPARQL & SQL)
File storage for larger binary data
– no size limits, any format,
difficult to query data
Archived datasets
HDF5 strategy for longer term
storage – compressed format with
embedded metadata
12](https://image.slidesharecdn.com/13-1345mograber-130624055022-phpapp02/85/The-Seven-Main-Challenges-of-an-Early-Warning-System-Architecture-12-320.jpg)
![©
Fraunhofer
13
ISCRAM 2013, Baden-Baden, Germany
4) Scalable and high performance storage of large distributed
datasets (cont.)
Discussion
Triple stores – good metadata query
support, poor scalability, standard
SPARQL
Relational databases – fast, ridgid
structure, poor metadata query support,
standard SQL
NoSQL type solutions [column stores etc.]
– suitable for our proposed hybrid
solution
Currently work on Apache Cassandra
integration
Map Reduce solutions – good for
distributed processing, efficient for
very large datasets, complex to setup
[overkill for real-time data with short
processing time horizons] 13](https://image.slidesharecdn.com/13-1345mograber-130624055022-phpapp02/85/The-Seven-Main-Challenges-of-an-Early-Warning-System-Architecture-13-320.jpg)
The Seven Main Challenges of an Early Warning System Architecture
- 1. © Fraunhofer 1 ISCRAM 2013, Baden-Baden, Germany The Seven Main Challenges of an Early Warning System Architecture J. Moßgraber, F. Chaves(1), S. Middleton, Z. Zlatev(2), R. Tao(3) (1) Fraunhofer Institute of Optronics, System Technologies and Image Exploitation (IOSB), Germany (2) IT Innovation Centre, Southampton, UK (3) Department of Electronic Engineering, Queen Mary University of London, UK
- 2. © Fraunhofer 2 ISCRAM 2013, Baden-Baden, Germany Sensors Decision Magic Early Warning System Architecture
- 3. © Fraunhofer 3 ISCRAM 2013, Baden-Baden, Germany Downhole Drilling Purpose Exploration & exploitation of oil & gas Retrieval of geothermal energy Controlled disposal of carbon dioxide Scientific drilling Safety Constraints Prevent crew, equipment and environment from injury, damage and pollution Prevent Crises
- 4. © Fraunhofer 4 ISCRAM 2013, Baden-Baden, Germany Tsunami Warning Systems
- 5. © Fraunhofer 5 ISCRAM 2013, Baden-Baden, Germany System-of-Systems Operational / managerial Independence of the Elements Different governments and institutions like Warning Centres, Task Forces, Scientific Institutions, Data Centres, … Evolutionary Development Integration of new Sensor networks, analysis algorithms, .. Emergent Behaviour Combines the knowledge of parts Geographic Distribution Tsunami Early Warning System for the Euro-Mediterranean area (> 20 national and at least one regional centre).
- 6. © Fraunhofer 6 ISCRAM 2013, Baden-Baden, Germany System-of-Systems (cont.) Communication is the key! 1st site Broker Cluster 2nd site Broker Cluster 3rd site Broker Cluster
- 7. © Fraunhofer 7 ISCRAM 2013, Baden-Baden, Germany The 7 Challenges 1.Build a scalable communication layer for a SoS 2.Build a resilient communication layer for a SoS 3.Efficiently publish large volumes of semantically rich sensor data 4.Scalable and high performance storage of large distributed datasets 5.Handling federated multi-domain heterogeneous data 6.Discovery of resources Sensors Decision Magic
- 8. © Fraunhofer 8 ISCRAM 2013, Baden-Baden, Germany 1) Build a scalable communication layer Requirements Open, heterogeneity, standard, language/OS independent Design decision Scalable communication using a Message- oriented Middleware (MOM): m-m Single MOM technology (Apache QPID): supports Advanced Message Queuing Protocol (AMQP) Discussion API standard, e.g., JMS / not a wireline standard, i.e., AMQP as supported by Qpid Text-based protocols, e.g., STOMP / not a binary protocol that is ↑ scalable Point-point systems, e.g., P2P, SOA / not generic M-M pub-sub model Proprietary MOMs / Not open-source, more difficult to enhance & instrument to monitor 8
- 9. © Fraunhofer 9 ISCRAM 2013, Baden-Baden, Germany 2) Build a resilient communication layer Requirements Broker Failure, e.g., broker crash due to underlay system failure Link Failure, e.g., low QoS in comms links or link broken Client Failure, e.g., subscriber client failed Design decision Broker Mirroring, i.e., messages are replicated to both primary broker and mirror broker Overlay Routing, i.e., messages are auto- switched to another overlay routing path according to the link status Durable Queue is selected to temporary stored the messages when sub has failed. Discussion SOA, distributed stream processing: have no inherent resilience 9
- 10. © Fraunhofer 10 ISCRAM 2013, Baden-Baden, Germany Twitter Crawler Unconventional / Human Sensors Source: http://xkcd.
- 11. © Fraunhofer 11 ISCRAM 2013, Baden-Baden, Germany 3) Efficiently publish large volumes of semantically rich sensor data Design decision Publish data to MOM & metadata to a semantic registry - fast, expressive metadata Combine different approaches (there is no „one size fits all“) SWE O&M XML messages with data embedded in them – slow, expressive metadata Support existing formats: WITS0 JSON encoded messages – fast, limited metadata Binary HDF5 / netCDF – fastest, limited metadata Discussion Database query via SSH tunnel – tight coupling of machines, SQL security issues HTTP or SQL request over MOM - QPID does not easily support this, SQL security issues 11
- 12. © Fraunhofer 12 ISCRAM 2013, Baden-Baden, Germany 4) Scalable and high performance storage of large distributed datasets Design decision Working datasets Hybrid databases [OWLIM for metadata, MySQL for data] – semantically rich and efficient queries, clients must understand two protocols (SPARQL & SQL) File storage for larger binary data – no size limits, any format, difficult to query data Archived datasets HDF5 strategy for longer term storage – compressed format with embedded metadata 12
- 13. © Fraunhofer 13 ISCRAM 2013, Baden-Baden, Germany 4) Scalable and high performance storage of large distributed datasets (cont.) Discussion Triple stores – good metadata query support, poor scalability, standard SPARQL Relational databases – fast, ridgid structure, poor metadata query support, standard SQL NoSQL type solutions [column stores etc.] – suitable for our proposed hybrid solution Currently work on Apache Cassandra integration Map Reduce solutions – good for distributed processing, efficient for very large datasets, complex to setup [overkill for real-time data with short processing time horizons] 13
- 14. © Fraunhofer 14 ISCRAM 2013, Baden-Baden, Germany 5) Handling federated multi-domain heterogeneous data Design decision Broker pattern for access / transformation of data between domains – scalable, slow adding an extra „hop‟ to workflow Domain ontologies in semantic registries – scalable, requires a lookup step Federated data queries – scalable, extra aggregation work for clients Discussion Data sources and/or apps map data to a global ontology – efficient, not practical to get agreement between domains Data sources and/or apps locally map between data models – does not scale, inconsistencies likely Automatic ontology alignment services – scalable, difficult & error prone, slow adding an extra „hop‟ to workflow 14
- 15. © Fraunhofer 15 ISCRAM 2013, Baden-Baden, Germany 6) Discovery of resources in a geo-distributed SoS Design decision Multiple semantic registries hosted by stakeholders – scalable, modular Separation of frontend/s and ontology store/s Shared ontology core (classes, relations, attributes, design patterns based on SSNO), registry-specific sub-classes and individuals – flexibility for adaptation to different domains Data and services described in semantically rich ways, allowing search/browse by both humans and machines – multiple interfaces for established protocols/standards15
- 16. © Fraunhofer 16 ISCRAM 2013, Baden-Baden, Germany 6) Discovery of resources in a geo-distributed SoS (cont.) Discussion Classical search-engines and catalogues – many out-of-the-box solutions, limited “semantic” search- capabilities, Monolithic systems – optimized and good performance for some types of applications, many additional tools but dependency between components One central semantic registry – no “synchronisation” of ontology core necessary, bottleneck if central registry not available 16
- 17. © Fraunhofer 17 ISCRAM 2013, Baden-Baden, Germany 7) Coordination of work between geo-distributed systems Design decision Decision tables & authoring tool for end users – self-documenting rule-sets and intuitive easy-to-use interfaces for non IT-experts, many available tools; comfortable mapping of “models” (sets of variables and rules) to ontology elements not yet standardized/available, Workflow engine(s) coordinating federated – message-based event processing for complex and rich choreographies, requirement-specific flexible adaptation of (standard) workflows Discussion Ontology reasoning – powerful and expressive reasoning und rule systems; require high-level of expertise, performance and (quality of) results depend on size/complexity of ontology, persistent storage of big ontologies still a problem. 17
- 18. © Fraunhofer 18 ISCRAM 2013, Baden-Baden, Germany System-of-systems 1st site Broker Cluster 2nd site Broker Cluster 3rd site Broker Cluster
- 19. © Fraunhofer 19 ISCRAM 2013, Baden-Baden, Germany Data Source(s) 1st site MOM Data Source(s) Data Source(s) Feeder Storage Historic DataCached Data Semantic Registry Workflow Service Data Source(s) Data Source(s) Processing Service R Receive realtime data R Get cached data and parameters; write results User Interface R Cache / store data Query R Steers R Receive notifications R Invoke & handle results RR Downstream Dissemination R R Register & request topic Generic TRIDEC Architecture
- 20. © Fraunhofer 20 ISCRAM 2013, Baden-Baden, Germany Example Workflow for Tsunami Early Warning
- 21. © Fraunhofer 21 ISCRAM 2013, Baden-Baden, Germany Data Source(s) 1st site MOM Data Source(s) Data Source(s) Feeder Storage Historic DataCached Data Semantic Registry Workflow Service Data Source(s) Data Source(s) Processing Service R Receive realtime data R Get cached data and parameters; write results User Interface R Cache / store data Query R Steers R Receive notifications R Invoke & handle results RR Downstream Dissemination R R Register & request topic Generic TRIDEC Architecture
- 22. © Fraunhofer 22 ISCRAM 2013, Baden-Baden, Germany Does it work? Tsunami TRIDEC software deployed at Instituto Português do Mar e da Atmosfera (IPMA) in Lissabon and the Kandilli Observatory and Earthquake Research Institute (KOERI) in Istanbul for testing purposes Two scenarios in the European-wide Tsunami exercise NEAMWave2012 were successfully validated New functionalities “Centre- to-Centre communication” via software systems between Turkey and Portugal and eyewitness reports sent from mobile devices via apps were available for the first time.
- 23. © Fraunhofer 23 ISCRAM 2013, Baden-Baden, Germany Conclusion and Future Work Main problems when dealing with the architecture of a SoS presented requires a scalable and resilient communication layer large amounts of data need to be published and processed requires a scalable storage concept respect the geo-distributed nature Future Work improving the resilience and workload allocation of the MOM improve the resilience of the Semantic Registry by providing a replication mechanism research the federated access to Big Data
- 24. © Fraunhofer 24 ISCRAM 2013, Baden-Baden, Germany Acknowledgements The presented work is done in collaboration with consortium of the TRIDEC project which is supported by the European Commission under the 7th Framework Programme (ICT-2009.4.3 Intelligent Information Management Project Reference: 258723)