Conf42 Python_ ML Enhanced Event Streaming Apps with Python Microservices

ML Enhanced Event
Streaming Apps with
Python Microservices
Tim Spann
Developer Advocate

2
Tim Spann
Principal Developer
Advocate
FLiP(N)/FLaNK Stack = Flink, Kafka, Pulsar and NiFi Stack
Streaming Systems & Data Architecture Expert
Experience:
● 15+ years of experience with streaming technologies
including NiFi, Kafka, Pulsar, Flink, Spark, Big Data,
Cloud, ML, IoT, Spring, Java, SQL, Python and more.
● Today, he helps to grow the Pulsar community
sharing rich technical knowledge and experience at
both global conferences and through individual
conversations.

https://bit.ly/32dAJft
FLiP Stack Weekly
This week in Apache Flink, Apache
Pulsar, Apache NiFi, Apache Spark and
open source friends.

4
Building
Real-Time
Requires a Team

Install Python 3 Pulsar Client
pip3 install pulsar-client=='2.11.0[all]'
# Depending on Platform May Need C++ Client Built
For Python on Pulsar on Pi https://github.com/tspannhw/PulsarOnRaspberryPi
https://pulsar.apache.org/docs/en/client-libraries-python/
6

Building a Python3 Producer
import pulsar
client = pulsar.Client('pulsar://localhost:6650')
producer
client.create_producer('persistent://conf/ete/first')
producer.send(('Simple Text Message').encode('utf-8'))
client.close()
7

python3 prod.py -su pulsar+ssl://name1.name2.snio.cloud:6651 -t
persistent://public/default/pyth --auth-params
'{"issuer_url":"https://auth.streamnative.cloud", "private_key":"my.json",
"audience":"urn:sn:pulsar:name:myclustr"}'
from pulsar import Client, AuthenticationOauth2
parse = argparse.ArgumentParser(prog=prod.py')
parse.add_argument('-su', '--service-url', dest='service_url', type=str,
required=True)
args = parse.parse_args()
client = pulsar.Client(args.service_url,
authentication=AuthenticationOauth2(args.auth_params))
https://github.com/streamnative/examples/blob/master/cloud/python/OAuth2Producer.py
https://github.com/tspannhw/FLiP-Pi-BreakoutGarden
Producer with OAuth to Cloud
8

import pulsar
from pulsar.schema import *
from pulsar.schema import AvroSchema
class thermal(Record):
uuid = String()
client = pulsar.Client('pulsar://pulsar1:6650')
thermalschema = AvroSchema(thermal)
producer =
client.create_producer(topic='persistent://public/default/pi-thermal-avro',
schema=thermalschema,properties={"producer-name": "thrm" })
thermalRec = thermal()
thermalRec.uuid = "unique-name"
producer.send(thermalRec,partition_key=uniqueid)
https://github.com/tspannhw/FLiP-Pi-Thermal
Example Avro Schema Usage

import pulsar
from pulsar.schema import JsonSchema
class weather(Record):
uuid = String()
client = pulsar.Client('pulsar://pulsar1:6650')
wsc = JsonSchema(thermal)
producer =
client.create_producer(topic='persistent://public/default/wthr,schema=wsc,pro
perties={"producer-name": "wthr" })
weatherRec = weather()
weatherRec.uuid = "unique-name"
producer.send(weatherRec,partition_key=uniqueid)
https://github.com/tspannhw/FLiP-Pi-Weather
https://github.com/tspannhw/FLiP-PulsarDevPython101
Example JSON Schema Usage

import pulsar
consumer =
client.subscribe('persistent://conf/ete/first',subscription_name='mine')
while True:
msg = consumer.receive()
print("Received message: '%s'" % msg.data())
consumer.acknowledge(msg)
client.close()
Building a Python Producer
11

pip3 install paho-mqtt
import paho.mqtt.client as mqtt
client = mqtt.Client("rpi4-iot")
row = { }
row['gasKO'] = str(readings)
json_string = json.dumps(row)
json_string = json_string.strip()
client.connect("pulsar-server.com", 1883, 180)
client.publish("persistent://public/default/mqtt-2",
payload=json_string,qos=0,retain=True)
https://www.slideshare.net/bunkertor/data-minutes-2-apache-pulsar-with-mqtt-for-edge-computing-lightning-2022
Sending MQTT Messages

pip3 install websocket-client
import websocket, base64, json
topic = 'ws://server:8080/ws/v2/producer/persistent/public/default/topic1'
ws = websocket.create_connection(topic)
message = "Hello Philly ETE Conference"
message_bytes = message.encode('ascii')
base64_bytes = base64.b64encode(message_bytes)
base64_message = base64_bytes.decode('ascii')
ws.send(json.dumps({'payload' : base64_message,'properties': {'device' :
'macbook'},'context' : 5}))
response = json.loads(ws.recv())
https://pulsar.apache.org/docs/en/client-libraries-websocket/
https://github.com/tspannhw/FLiP-IoT/blob/main/wspulsar.py
https://github.com/tspannhw/FLiP-IoT/blob/main/wsreader.py
Sending Websocket Messages

pip3 install kafka-python
from kafka import KafkaProducer
from kafka.errors import KafkaError
row = { }
row['gasKO'] = str(readings)
json_string = json.dumps(row)
json_string = json_string.strip()
producer = KafkaProducer(bootstrap_servers='pulsar1:9092',retries=3)
producer.send('topic-kafka-1', json.dumps(row).encode('utf-8'))
producer.flush()
https://github.com/streamnative/kop
https://docs.streamnative.io/platform/v1.0.0/concepts/kop-concepts
Sending Kafka Messages

bin/pulsar-admin functions create --auto-ack true --py py/src/sentiment.py
--classname "sentiment.Chat" --inputs "persistent://public/default/chat"
--log-topic "persistent://public/default/logs" --name Chat --output
"persistent://public/default/chatresult"
https://github.com/tspannhw/pulsar-pychat-function
DevOps: Deploying Functions
15

17
Example Walk Through
import pulsar
class Stock (Record):
symbol = String()
ts = Float()
currentts = Float()
volume = Float()
price = Float()
tradeconditions = String()
uuid = String()

18
producer = client.create_producer(topic='persistent://public/default/stocks'
,schema=JsonSchema(Stock),properties={"producer-name":
"py-stocks","producer-id": "pystocks1" })

19
uuid_key =
'{0}_{1}'.format(strftime("%Y%m%d%H%M%S",gmtime()),uuid.uuid4())
stockRecord = Stock()
stockRecord.symbol = stockitem['s']
stockRecord.ts = float(stockitem['t'])
stockRecord.currentts = float(strftime("%Y%m%d%H%M%S",gmtime()))
stockRecord.volume = float(stockitem['v'])
stockRecord.price = float(stockitem['p'])
stockRecord.tradeconditions = ','.join(stockitem['c'])
stockRecord.uuid = uuid_key
if ( stockitem['s'] != '' ):
producer.send(stockRecord,partition_key=str(uuid_key))

22
Example Web Page
https://github.com/tspannhw/pulsar-transit-function
<link rel="stylesheet" type="text/css"
href="https://cdn.datatables.net/1.11.3/css/jquery.dataTables.min.css">
<script type="text/javascript" language="javascript"
src="https://code.jquery.com/jquery-3.5.1.js">
</script>
<script type="text/javascript" language="javascript"
src="https://cdn.datatables.net/1.11.3/js/jquery.dataTables.min.js">
</script>

23
<table id="example" class="display" style="width:100%">
<thead>
<tr>
<th><b>Key</b></th>
<th><b>Publish Time</b></th>
<th><b>Msg</b></th>
<th><b>Latitude</b></th>
<th><b>Longitude</b></th>
<th><b>Pub Date</b></th>
<th><b>System Time</b></th> </tr> </thead>
<tfoot>
<tr>
<th><b>Key</b></th>
<th><b>Publish Time</b></th>
<th><b>Msg</b></th>
<th><b>Latitude</b></th>
<th><b>Longitude</b></th>
<th><b>Pub Date</b></th>
<th><b>System Time</b></th>
</tr>
</tfoot>
</table>

24
$(document).ready(function() {
var t = $('#example').DataTable();
var wsUri =
"ws://pulsar1:8080/ws/v2/consumer/persistent/public/default/trans
com/tc-reader?subscriptionType=Shared";
websocket = new WebSocket(wsUri);
websocket.onopen = function(evt) {
console.log('open');
};
websocket.onerror = function(evt) {console.log('ERR', evt)};

25
websocket.onmessage = function(evt) {
var dataPoints = JSON.parse(evt.data);
if ( dataPoints === undefined || dataPoints == null || dataPoints.payload === undefined || dataPoints.payload
== null ) {
return;
}
if (websocket.readyState === WebSocket.OPEN) {
websocket.send("{"messageId": "" + dataPoints.messageId + ""}");
}
if (IsJsonString(atob(dataPoints.payload))) {
var pulsarMessage = JSON.parse(atob(dataPoints.payload));
if ( pulsarMessage === undefined || pulsarMessage == null ) {
return;
}
t.row.add( [ dataPoints.key, dataPoints.publishTime, pulsarMessage.title,
pulsarMessage.latitude, pulsarMessage.longitude, pulsarMessage.pubDate,
pulsarMessage.ts] ).draw(true );
}
};
} );
https://medium.com/@tspann/transit-watch-real-time-feeds-d98ff62b3bbb

• Guaranteed delivery
• Data buffering
- Backpressure
- Pressure release
• Prioritized queuing
• Flow specific QoS
- Latency vs. throughput
- Loss tolerance
• Data provenance
• Supports push and pull
models
• Hundreds of processors
• Visual command and
control
• Over a sixty sources
• Flow templates
• Pluggable/multi-role
security
• Designed for extension
• Clustering
• Version Control
DataFlows for Data Ingest, Movement and Routing

• Moving Binary, Unstructured, Image
and Tabular Data
• Enrichment
• Universal Visual Processor
• Simple Event Processor
• Routing
• Feeding data to Central Messaging
• Support for modern protocols
• Kafka Protocol Source/Sink
• Pulsar Protocol Source/Sink
The Power of Apache NiFi

Architecture
https://nifi.apache.org/docs/nifi-docs/html/overview.html

29
https://github.com/tspannhw https://www.datainmotion.dev/
https://www.meetup.com/futureofdata-princeton/

Conf42 Python_ ML Enhanced Event Streaming Apps with Python Microservices

More Related Content

Similar to Conf42 Python_ ML Enhanced Event Streaming Apps with Python Microservices (20)

More from Timothy Spann (20)

Recently uploaded (20)

Conf42 Python_ ML Enhanced Event Streaming Apps with Python Microservices