Using Kafka in your python application - Python fwdays 2020

Using Kafka in your
python application
Oleksandr Tarasenko
EVO Pay / EVO.company

Agenda
● intro to kafka
● kafka and other solutions
● kafka + python
● EVO solutions with kafka
4

What is Apache Kafka
A Distributed Streaming Platform
● Publish & Subscribe
● Process streams
● Store data
6

What is Apache Kafka
A Distributed Streaming Platform
● Publish & Subscribe
● Process streams
● Store data
● Queues ?
8

Kafka vs ???
● kafka vs rabbitmq
● kafka vs nsq
● kafka vs jms
● kafka vs nuts
● kafka vs pulsar
● kafka vs rest ?
● kafka vs akka ??
12

Apache Kafka good in
● Event sourcing
● Stream processing
● Messaging
● Log aggregation
● Activity tracking
● ...
14

Kafka Ecosystem
https://cwiki.apache.org/confluence/display/KAFKA/Ecosystem

Kafka Ecosystem
https://cwiki.apache.org/confluence/display/KAFKA/Ecosystem
Stream processing
Packaging
Distributions
Kafka connect
Hadoop integration
Database integration
Metrics
Management consoles
Logging
AWS integration
Kafka Camel Integration
Packing and Deployment

Kafka in python world
● kafka-python
● confluent-kafka-python
● pykafka
● aiokafka
● faust*
18

Kafka in python world
● kafka-python (sync)
● confluent-kafka-python (sync)
● pykafka (dead)
● aiokafka (async)
● faust* (async)
19

kafka-python
https://github.com/dpkp/kafka-python
https://kafka-python.readthedocs.io/
● pythonic
● work with old kafka versions

kafka-python
from kafka import KafkaProducer
producer = KafkaProducer(bootstrap_servers='localhost:1234')
...
for product_data in some_data_source:
producer.send('product_topic', product_data)
future = producer.send('product_topic', b'product_data')
result = future.get(timeout=60)
producer.flush()
…
producer.send('product_topic', key=b'product_category_id', value=b'product_data')
21

kafka-python
from kafka import KafkaConsumer
consumer = KafkaConsumer('product_topic')
for msg in consumer:
print(msg)
...
consumer = KafkaConsumer('product_topic', group_id='discount_product_group')
for msg in consumer:
print(msg)
22

confluent-kafka-python
https://github.com/confluentinc/confluent-kafka-python
https://docs.confluent.io/current/clients/confluent-kafka-python/index.html
● librdkafka
● faster
● thread safe

from confluent_kafka import Producer
p = Producer({'bootstrap.servers': 'localhost:1234'})
def delivery_report(err, msg):
if err is not None:
print('Message delivery failed: {}'.format(err))
else:
print('Message delivered to {} [{}]'.format(msg.topic(), msg.partition()))
for data in some_data_source:
p.poll(0)
p.produce('product_topic', data.encode('utf-8'), callback=delivery_report)
p.flush()
24

from confluent_kafka import Consumer
c = Consumer({
'bootstrap.servers': 'localhost:1234',
'group.id': 'discount_product_group',
'auto.offset.reset': 'earliest'
})
c.subscribe(['product_topic'])
while True:
msg = c.poll(1.0)
if msg is None:
continue
if msg.error():
print("Consumer error: {}".format(msg.error()))
continue
print('Received message: {}'.format(msg.value().decode('utf-8')))
c.close() 25

aiokafka
https://github.com/aio-libs/aiokafka
https://aiokafka.readthedocs.io/en/stable/
● aio-libs
● async/await model
● based on kafka-python
● ukrainian developers

aiokafka
from aiokafka import AIOKafkaProducer
import asyncio
loop = asyncio.get_event_loop ()
async def send_one():
producer = AIOKafkaProducer (
loop=loop, bootstrap_servers ='localhost:1234' )
await producer.start()
try:
await producer.send_and_wait("product_topic" , b"product_data" )
finally:
await producer.stop()
loop.run_until_complete (send_one())
27

aiokafka
from aiokafka import AIOKafkaConsumer
import asyncio
async def consume():
consumer = AIOKafkaConsumer (
'product_topic' , 'other_topic',
loop=loop, bootstrap_servers ='localhost:1234' ,
group_id ="discount_product_group" )
await consumer.start()
try:
async for msg in consumer:
print("consumed: ", msg.topic, msg.partition, msg.offset,
msg.key, msg.value, msg.timestamp)
finally:
await consumer.stop()
loop.run_until_complete (consume())
28

faust
https://github.com/robinhood/faust
https://faust.readthedocs.io/en/latest/
● python 3.6+, async/await
● like framework
● standalone
● kafka streams
● celery*

faust
import faust
app = faust.App('myapp', broker='kafka://localhost:1234')
class Product(faust.Record):
product_id: str
price: int
@app.agent(value_type=Product)
async def product(products):
async for product in products:
print(f'Product for {product.product_id}: {product.price}')
30

faust
import faust
class Product(faust.Record):
product_id: str
price: int
app = faust.App('hello-app', broker='kafka://localhost:1234')
topic = app.topic('hello-topic', value_type=Product)
@app.agent(topic)
async def example_source(products):
async for product in products:
print(f'Hello from {product.product_id} to {product.price}')
@app.timer(value_type=1.0)
async def example_producer(app):
await example_source.send(
value=Product(product_id='123zxc', price=100),
)
if __name__ == '__main__':
app.main()
31

Example 1 - Products indexation

Example scheme with python-kafka

Example 2 - Delivery data stream

Example scheme with aiokafka + faust

Using Kafka in your python application - Python fwdays 2020

More Related Content

What's hot (20)

Similar to Using Kafka in your python application - Python fwdays 2020 (20)

Recently uploaded (20)

Using Kafka in your python application - Python fwdays 2020