Big Data and Machine Learning with FIWARE
0 likes569 views
The document provides an overview of big data and machine learning within the FIWARE framework, covering topics such as Apache Flink and Spark, ETL architecture, and machine learning algorithms. It details the integration of FIWARE components, including the Orion Context Broker, with big data tools and addresses the development lifecycle of machine learning. Additionally, it includes references for further reading and resources related to FIWARE and its connectors.
![Basic example
17
final val URL_CB = "http://flinkexample_orion_1:1026/v2/entities/"
final val CONTENT_TYPE = ContentType.JSON
final val METHOD = HTTPMethod.POST
def main(args: Array[String]): Unit = {
val env = StreamExecutionEnvironment.getExecutionEnvironment
// Create Orion Source. Receive notifications on port 9001
val eventStream = env.addSource(new OrionSource(9001))
// Process event stream](https://image.slidesharecdn.com/session9-bigdataandmachinelearningwithfiware-191127084209/85/Big-Data-and-Machine-Learning-with-FIWARE-18-320.jpg)
![Basic example
18
// Process event stream
val processedDataStream = eventStream
.flatMap(event => event.entities)
.map(entity => {
val temp = entity.attrs("temperature").value.asInstanceOf[Number].floatValue()
new Temp_Node(entity.id, temp)
})
.keyBy("id")
.timeWindow(Time.seconds(5), Time.seconds(2))
.min("temperature")
.map(tempNode => {
val url = URL_CB + tempNode.id + "/attrs"
OrionSinkObject(tempNode.toString, url, CONTENT_TYPE, METHOD)
})
// Add Orion Sink](https://image.slidesharecdn.com/session9-bigdataandmachinelearningwithfiware-191127084209/85/Big-Data-and-Machine-Learning-with-FIWARE-19-320.jpg)
Big Data and Machine Learning with FIWARE
- 1. Session 9 - Big Data and Machine Learning with FIWARE Fernando López, Cloud & Platform Senior Expert [email protected] @flopezaguilar FIWARE Foundation, e.V.
- 2. Learning Goals 1 ● Introduction to Big Data ● Different between Apache Flink and Spark ● FIWARE connectors ● (Work in Progress) Machine Learning in FIWARE
- 3. 2 Introduction to Big Data
- 4. Big Data 3
- 5. Big Data Analytics 4 Indexed Storage (RDBMS, Apache Solr) Interactive Processing (e.g. Drill, BigQuery, OLAP) MapReduce (e,g, Spark, Hadoop) Realtime Analytics (CEP, Stream Processing) In-Memory Computing (e.g. Spark, SAP Hana, VoltDB)SizeodtheDataHandled (persecond) millis seconds minutes hours days Time to Act 100k events (100MBs) 1k events (1MBs) 100 events (10KBs)
- 9. 8 Kappa architecture Source: Siddharth Mittal
- 10. Simple Smart solutions: Reference Architecture 9 Draco Kurento Wirecloud QuantumLeap Knowage Flink CrateDB
- 11. 10 FIWARE Cosmos: Orion Flink Connector
- 12. Features ▪ The Cosmos Generic Enabler enables an easier BigData analysis over context integrated with some of the most popular BigData platforms. ▪ Batch Processing ▪ Stream Processing (Real-time) ▪ Direct data ingestion ▪ Direct connection with Context Broker ▪ Multiple Sinks 11
- 13. Apache Flink ▪ Framework and distributed processing engine for stateful computations over unbounded and bounded data streams. ▪ Designed to run in all common cluster environments, perform computations at in-memory speed and at any scale. 12
- 14. Architecture 13
- 15. Connection 14 ORION Context Broker Flink Cluster Flink Job (JAR) orion-flink-connector HTTP POST (Notification) HTTP POST/PUT/PATCH OrionSource OrionSink
- 16. OrionSource 15 ▪ Receives data from the Orion Context Broker from a given port. ▪ The received data is a Stream of NgsiEvent object. val eventStream = env.addSource(new OrionSource(9001))
- 17. OrionSink 16 ▪ Sends data back to the Orion Context Broker: ▪ Takes a stream of OrionSinkObjects as a source: • content: Message content in String format. If it is a JSON, it needs to be stringified. • url: URL to which the message should be sent • contentType: Type of HTTP content of the message (JSON, Plain) • method: HTTP method of the message (POST, PUT, PATCH) OrionSink.addSink( processedDataStream )
- 18. Basic example 17 final val URL_CB = "http://flinkexample_orion_1:1026/v2/entities/" final val CONTENT_TYPE = ContentType.JSON final val METHOD = HTTPMethod.POST def main(args: Array[String]): Unit = { val env = StreamExecutionEnvironment.getExecutionEnvironment // Create Orion Source. Receive notifications on port 9001 val eventStream = env.addSource(new OrionSource(9001)) // Process event stream
- 19. Basic example 18 // Process event stream val processedDataStream = eventStream .flatMap(event => event.entities) .map(entity => { val temp = entity.attrs("temperature").value.asInstanceOf[Number].floatValue() new Temp_Node(entity.id, temp) }) .keyBy("id") .timeWindow(Time.seconds(5), Time.seconds(2)) .min("temperature") .map(tempNode => { val url = URL_CB + tempNode.id + "/attrs" OrionSinkObject(tempNode.toString, url, CONTENT_TYPE, METHOD) }) // Add Orion Sink
- 20. Basic example 19 // Add Orion Sink OrionSink.addSink( processedDataStream ) // … }
- 21. 20 FIWARE Cosmos: Orion Spark Connector
- 23. Spark Scheduler 22 join union groupBy map Stage 3 Stage 1 Stage 2 A: B: C: D: E: F: G: = cached data partition ▪ Dryad-like DAGs ▪ Pipelines functions within a stage ▪ Cache-aware work reuse & locality ▪ Partitioning-aware to avoid shuffles
- 24. Motivation of Spark 23 ▪ Iterative algorithms (machine learning, graphs) ▪ Interactive data mining tools (R, Excel, Python)
- 25. Connection 24 ORION Context Broker Spark Cluster Spark Job (JAR) orion-spark-connector HTTP POST (Notification) HTTP POST/PUT/PATCH OrionReceiver OrionSink
- 26. 25 FIWARE & Machine Learning
- 28. Machine Learning Development Lifecycle 27
- 29. Machine Learning Algorithms 28 ▪ Some solutions have high algorithm complexity ▪ Some can be parallelized in a cluster (FlinkML) ▪ Other can use GPU (e.g. Tensorflow) ▪ Even each case could be different we try to set up some generic life cycle.
- 30. 29 ML Standard Solution ▪ Each problem requires an analysis of which ML algorithm suits our data. ▪ Later, the training dataset needs to be set up. ▪ Each problem may be slightly different (“same same but different”). ▪ We can provide some solutions for some cases and use a proper dataset. ▪ The tool to use (Spark, Flink, Tensorflow) depends on the chosen ML algorithm (not all the ML algorithms are in all the architectures).
- 31. 30 Current Status Orion Connector Orion Source/Receiver + Orion Sink ✔ ✔ RTD Documentation ✔ ✔ Unit Tests ✔ ✔ Examples ✔ ✔ Step-by-step tutorial ✔ Support NGSI LD
- 32. Summary: Terms 31 ● OLAP, Online Analytical Processing. ● OLTP, Online Transaction Processing. ● RDBMS, Relational Database Management System. ● ETL, Extract, Transform, Load. ● ERP, Enterprise Resource Planning. ● CRM, Customer relationship management.
- 33. Summary: Terms 32 ● OSV, Output Slot Vector. ● BI, Business Intelligence. ● HDFS, Hadoop Distributed File System ● DAG, Directed Acyclic Graph. The DAG defines the dataflow of the application, and the vertices of the graph defines the operations that are to be performed on the data.
- 34. References ▪ FIWARE Catalogue • https://www.fiware.org/developers/catalogue ▪ FIWARE Academy: • https://fiware-academy.readthedocs.io/en/latest/processing/wirecloud ▪ Installation, administration & reference documentation is available on Read The Docs: • https://fiware-cosmos-flink.readthedocs.io 33
- 35. References ▪ GitHub • https://github.com/ging/fiware-cosmos-orion-flink-connector • https://github.com/ging/fiware-cosmos-orion-spark-connector • https://github.com/ging/fiware-cosmos-orion-flink-connector-examples • https://github.com/ging/fiware-cosmos-orion-spark-connector-examples 34
- 38. 3 7