Flink Forward Berlin 2017: Zohar Mizrahi - Python Streaming API
1 like1,725 views
The document discusses a Python streaming API being developed for use with Flink, focusing on machine learning applications and real-time data processing. It covers the architecture, challenges, and performance considerations of integrating Python with Java-based systems such as Flink, along with a demonstration and example usage. Key features include the usability of Python libraries and ongoing development efforts to ensure API coverage and functionality.
![www.parallelm.com
Python Script - UDF
class Tokenizer(FlatMapFunction):
def flatMap(self, value, collector):
for word in value.lower().split():
collector.collect((1, word))
class Selector(KeySelector):
def getKey(self, input):
return input[1]
class Sum(ReduceFunction):
def reduce(self, input1, input2):
count1, word1 = input1
count2, word2 = input2
return (count1 + count2, word1)
12](https://image.slidesharecdn.com/pythonstreamingapi-170913122238/85/Flink-Forward-Berlin-2017-Zohar-Mizrahi-Python-Streaming-API-12-320.jpg)
Flink Forward Berlin 2017: Zohar Mizrahi - Python Streaming API
- 1. www.parallelm.com Python Streaming API 1 Zohar Mizrahi Senior Software Architect ParallelM Flink Forward 2017, Berlin, Germany
- 2. www.parallelm.com What Will Be Covered ● ParallelM ● Python in Machines Learning ● Python Batch API ● Python Streaming API ● Live Demo 2
- 3. www.parallelm.com ParallelM ParallelM accelerates time to value of AI initiatives by helping ML Ops and Data Science teams deploy and manage Machine Learning (ML) in Production We have put much effort in Flink, because of its exceptional design and ability to handle high speed real time and true stream processing 3
- 4. www.parallelm.com Python In Machine Learning ● Popular in data analysis (NumPy, SciPy, Matplotlib, Panda, etc.) ● Very easy to learn ● Very easy to read ● Does not require compilation ● Awesome online community 4
- 5. www.parallelm.com Python Batch Processing Overview 5 Client (JVM) Script (Python) HDFS JobManager (JVM) TaskManager (JVM) Script (Python) MMAP Files
- 6. www.parallelm.com … but Flink is a Stream Processing framework ... What about Python for Streaming Data? 6
- 7. www.parallelm.com Proposed Python Streaming API Architecture 7 Jython, Python Script Java Streaming API Client (JVM) HDFS JobManager (JVM) Jython, Python Script UDFs TaskManager (JVM) Python Streaming API Python Streaming API - Python naming - Serialization/Deseri alization - Thin layer
- 8. www.parallelm.com Jython ● Python engine in java (http://www.jython.org/) ● Possible to use CPython extensions like NumPy or SciPy with JyNI (http://jyni.org/) ● Glitches ○ The latest supported Python version is 2.7 ○ No official statement for coming support in Python 3.x 8
- 9. www.parallelm.com What Jython Challenges I had to solve? ● Java class serial ID mismatch ○ Execution of the same script multiple times ● Different namespaces for python classes with same name but different code ○ Execution of different scripts having the same class name ● Python paths and imports issues ○ A python script may import additional files and folder 9
- 10. www.parallelm.com Performance Considerations ● Initialisation of the Jython framework impose a fixed overhead of 2 ~ 5 seconds: ○ Client - whenever submitted ○ TaskManager - only once, on the first submitted job ● Java/Scala vs. Python ○ No high-scale tests were conducted yet ○ There’s a room for optimizations 10
- 11. www.parallelm.com Python Script - main def main(): env = PythonStreamExecutionEnvironment.get_execution_environment() env.read_text_file(“/tmp/book.txt”) .flat_map(Tokenizer()) .key_by(Selector()) .time_window(milliseconds(50)) .reduce(Sum()) .print() env.execute(True) 11
- 12. www.parallelm.com Python Script - UDF class Tokenizer(FlatMapFunction): def flatMap(self, value, collector): for word in value.lower().split(): collector.collect((1, word)) class Selector(KeySelector): def getKey(self, input): return input[1] class Sum(ReduceFunction): def reduce(self, input1, input2): count1, word1 = input1 count2, word2 = input2 return (count1 + count2, word1) 12
- 13. www.parallelm.com Status / API Coverage ● Pending pull request (#3838) ● New project under: flink-libraries/flink-streaming-python ● Partial coverage of the whole streaming API (Beta) 13
- 14. www.parallelm.com Tests / Examples ● Internal tests are under: flink-libraries/flink-streaming-python/src/test/python/org/a pache/flink/streaming/python/api ● One complete example: flink-examples/flink-examples-streaming/src/main/python/fibo nacci.py 14
- 15. www.parallelm.com > ./bin/pyflink-stream.sh /tmp/fibonacci.py - --local Notes: ● New command line tool: pyflink-stream.sh ● Command line arguments: after the dash(`-`) ● For local execution: env.execute(True) ● Cluster mode requires HDFS How to execute 15
- 16. www.parallelm.com 16 ● Fibonacci python example ● Functionality ○ Calculates fibonacci series up to an upper bound ● Input ○ “<x>, <y>” - stream of pairs of numbers ● Output ○ ((<x>, <y>), <#iters>) - the original pair along with the iterations number Demo Time