FIWARE Real-time Processing of Historic Context Information using Apache Flink. FIWARE Global Summit - Málaga 2018 - UPM Team
0 likes48 views
The document discusses the FIWARE Cosmos Orion Flink connector, which allows Apache Flink to interact with the Orion Context Broker for data stream processing. It explains the architecture of the connector, including the orionsource for receiving notifications and the orionsink for sending data back to the broker. The document also provides example code for processing event streams and applying functions like average and maximum temperature calculations.
![Basic Example
final val URL_CB = "http://flinkexample_orion_1:1026/v2/entities/"
final val CONTENT_TYPE = ContentType.JSON
final val METHOD = HTTPMethod.POST
def main(args: Array[String]): Unit = {
val env = StreamExecutionEnvironment.getExecutionEnvironment
// Create Orion Source. Receive notifications on port 9001
val eventStream = env.addSource(new OrionSource(9001))
// Process event stream
val processedDataStream = eventStream
.flatMap(event => event.entities)
.map(entity => {
val temp = entity.attrs("temperature").value.asInstanceOf[Number].floatValue()
new Temp_Node(entity.id, temp)
})
.keyBy("id")
.timeWindow(Time.seconds(5), Time.seconds(2))
.min("temperature")
.map(tempNode => {
val url = URL_CB + tempNode.id + "/attrs"
OrionSinkObject(tempNode.toString, url, CONTENT_TYPE, METHOD)
})
// Add Orion Sink
OrionSink.addSink( processedDataStream )
// …
}
9](https://image.slidesharecdn.com/9-upm-fiware-real-timeprocessingofhistoriccontextinformationusingapacheflink-181201093353/85/FIWARE-Real-time-Processing-of-Historic-Context-Information-using-Apache-Flink-FIWARE-Global-Summit-Malaga-2018-UPM-Team-9-320.jpg)
![def main(args: Array[String]): Unit = {
val env = StreamExecutionEnvironment.getExecutionEnvironment
// Create Orion Source. Receive notifications on port 9001
val eventStream = env.addSource(new OrionSource(9001))
// Process event stream
val processedDataStream = eventStream
.flatMap(event => event.entities)
.map(entity => {
val temp = entity.attrs("temperature").value.asInstanceOf[Number].floatValue()
new Temp_Node( entity.id, temp)
})
.keyBy("id")
.timeWindow(Time.seconds(10))
.aggregate(new Average)
// print the results with a single thread, rather than in parallel
processedDataStream.print().setParallelism(1)
env.execute("Socket Window NgsiEvent")
}
Demo: Average temperature for each entity
13](https://image.slidesharecdn.com/9-upm-fiware-real-timeprocessingofhistoriccontextinformationusingapacheflink-181201093353/85/FIWARE-Real-time-Processing-of-Historic-Context-Information-using-Apache-Flink-FIWARE-Global-Summit-Malaga-2018-UPM-Team-11-320.jpg)
![Demo: Maximum temperature overall
def main(args: Array[String]): Unit = {
val env = StreamExecutionEnvironment.getExecutionEnvironment
// Create Orion Source. Receive notifications on port 9001
val eventStream = env.addSource(new OrionSource(9001))
// Process event stream
val processedDataStream = eventStream
.flatMap(event => event.entities)
.map(entity => {
val temp = entity.attrs("temperature").value.asInstanceOf[Number].floatValue()
println("Entity",entity.id, "Temperature", temp)
new Temp_Node( entity.id, temp)
})
.timeWindowAll(Time.seconds(10) )
.max("temperature")
// print the results with a single thread, rather than in parallel
processedDataStream.print().setParallelism(1)
env.execute("Socket Window NgsiEvent")
}
15](https://image.slidesharecdn.com/9-upm-fiware-real-timeprocessingofhistoriccontextinformationusingapacheflink-181201093353/85/FIWARE-Real-time-Processing-of-Historic-Context-Information-using-Apache-Flink-FIWARE-Global-Summit-Malaga-2018-UPM-Team-12-320.jpg)
FIWARE Real-time Processing of Historic Context Information using Apache Flink. FIWARE Global Summit - Málaga 2018 - UPM Team
- 1. FIWARE Orion Flink Connector Sonsoles López ([email protected]) Andres Muñoz ([email protected]) Joaquin Salvachua ([email protected]) Gabriel Huecas ([email protected]) Universidad Politécnica de Madrid @jsalvachua, @FIWARE
- 2. Flink Apache Flink is a framework and distributed processing engine for stateful computations over unbounded and bounded data streams. Flink has been designed to run in all common cluster environments, perform computations at in-memory speed and at any scale. 2
- 5. fiware-cosmos-orion-flink-connector https://github.com/ging/fiware-cosmos-orion-flink-connector https://fiware-cosmos-flink.readthedocs.io Two parts: ● OrionSource: Receive notifications from the Context Broker ● OrionSink: Write data into the Context Broker 5
- 6. Connection 6 ORION Context Broker Flink Cluster Flink Job (JAR) orion-flink-connector HTTP POST (Notification) HTTP POST/PUT/PATCH OrionSource OrionSink
- 7. OrionSource Receives data from the Orion Context Broker from a given port. The received data is a Stream of NgsiEvent objects val eventStream = env.addSource(new OrionSource(9001)) 7
- 8. OrionSink Sends data back to the Orion Context Broker Takes a stream of OrionSinkObjects as a source: ● content: Message content in String format. If it is a JSON, it needs to be stringified ● url: URL to which the message should be sent ● contentType: Type of HTTP content of the message (JSON, Plain) ● method: HTTP method of the message (POST, PUT, PATCH) OrionSink.addSink( processedDataStream ) 8
- 9. Basic Example final val URL_CB = "http://flinkexample_orion_1:1026/v2/entities/" final val CONTENT_TYPE = ContentType.JSON final val METHOD = HTTPMethod.POST def main(args: Array[String]): Unit = { val env = StreamExecutionEnvironment.getExecutionEnvironment // Create Orion Source. Receive notifications on port 9001 val eventStream = env.addSource(new OrionSource(9001)) // Process event stream val processedDataStream = eventStream .flatMap(event => event.entities) .map(entity => { val temp = entity.attrs("temperature").value.asInstanceOf[Number].floatValue() new Temp_Node(entity.id, temp) }) .keyBy("id") .timeWindow(Time.seconds(5), Time.seconds(2)) .min("temperature") .map(tempNode => { val url = URL_CB + tempNode.id + "/attrs" OrionSinkObject(tempNode.toString, url, CONTENT_TYPE, METHOD) }) // Add Orion Sink OrionSink.addSink( processedDataStream ) // … } 9
- 10. Examples https://github.com/ging/fiware-cosmos-orion-flink-connector-examples https://fiware-cosmos-flink-examples.readthedocs.io ● Example 1: Simulated Orion Source Notification ● Example 2: Complete Orion Scenario with docker-compose ● Example 3: Packaging the code and submitting it to the Flink Job Manager ● Example 4: More complex operations (Flink AggregateFunction) ● Example 5: Structured values for attributes (objects) 10
- 11. def main(args: Array[String]): Unit = { val env = StreamExecutionEnvironment.getExecutionEnvironment // Create Orion Source. Receive notifications on port 9001 val eventStream = env.addSource(new OrionSource(9001)) // Process event stream val processedDataStream = eventStream .flatMap(event => event.entities) .map(entity => { val temp = entity.attrs("temperature").value.asInstanceOf[Number].floatValue() new Temp_Node( entity.id, temp) }) .keyBy("id") .timeWindow(Time.seconds(10)) .aggregate(new Average) // print the results with a single thread, rather than in parallel processedDataStream.print().setParallelism(1) env.execute("Socket Window NgsiEvent") } Demo: Average temperature for each entity 13
- 12. Demo: Maximum temperature overall def main(args: Array[String]): Unit = { val env = StreamExecutionEnvironment.getExecutionEnvironment // Create Orion Source. Receive notifications on port 9001 val eventStream = env.addSource(new OrionSource(9001)) // Process event stream val processedDataStream = eventStream .flatMap(event => event.entities) .map(entity => { val temp = entity.attrs("temperature").value.asInstanceOf[Number].floatValue() println("Entity",entity.id, "Temperature", temp) new Temp_Node( entity.id, temp) }) .timeWindowAll(Time.seconds(10) ) .max("temperature") // print the results with a single thread, rather than in parallel processedDataStream.print().setParallelism(1) env.execute("Socket Window NgsiEvent") } 15