Apache Flink London Meetup - Let's Talk ML on Flink
0 likes226 views
This document summarizes a presentation about serving machine learning models with Apache Flink. It discusses Flink's ML roadmap, modeling concepts, and model serving considerations like customer needs, governance, and the model lifecycle. It proposes using Flink's streaming capabilities to dynamically update models without interruption. A prototype exploits Flink joins to merge prediction requests with cached models. Next steps include finalizing specifications, integrating TensorFlow and PMML models, and addressing governance.
![Model representationOn the wire
syntax = "proto3";
// Description of the trained model.
message ModelDescriptor {
// Model name
string name = 1;
// Human readable description.
string description = 2;
// Data type for which this model is applied.
string dataType = 3;
// Model type
enum ModelType {
TENSORFLOW = 0;
TENSORFLOWSAVED = 2;
PMML = 2;
};
ModelType modeltype = 4;
oneof MessageContent {
// Byte array containing the model
bytes data = 5;
string location = 6;
}
} 18
Internal
trait Model {
def score(input : AnyVal) : AnyVal
def cleanup() : Unit
def toBytes() : Array[Byte]
def getType : Long
}
trait ModelFactoryl {
def create(input : ModelDescriptor) :
Model
def restore(bytes : Array[Byte]) : Model
}](https://image.slidesharecdn.com/flinkmeetuplondon-serving-171120004000/85/Apache-Flink-London-Meetup-Let-s-Talk-ML-on-Flink-18-320.jpg)
Apache Flink London Meetup - Let's Talk ML on Flink
- 1. Let’s Talk ML on Flink Boris Lublinsky Stavros Kontopoulos Lightbend Apache Flink London Meetup (15/11/2017)
- 2. Talk’s agenda • Flink ML Roadmap • Machine learning & Models • Model Serving with Flink • Results and next steps 2
- 3. Flink ML Flink ML Roadmap: https://docs.google.com/document/d/1afQbvZBTV15qF3vobVWUjxQc49h3 Ud06MIRhahtJ6dw Model Serving Online Learning Incremental learning Batch api (https://issues.apache.org/jira/browse/FLINK-2396)? 3
- 4. How people see ML 4
- 5. Reality 5
- 6. What is the Model? 6 A model is a function transforming inputs to outputs - y = f(x),for example: Linear regression: y = ac + a1*x1 + … + an*xn Neural network: Such a definition of the model allows for an easy implementation of model’s composition. From the implementation point of view it is just function composition
- 7. Model learning pipeline UC Berkeley AMPLab introduced machine learning pipelines as a graph defining the complete chain of data transformation. 7
- 8. Models proliferation Data Scientist Software engineer 8
- 10. Model lifecycle considerations • Models tend to change • Update frequencies vary greatly - from hourly to quarterly/yearly • Model version tracking • Model release practices • Model update process 10
- 11. Customer SLA • Response time - time to calculate prediction • Throughput - predictions per second • Support for running multiple models • Model update - how quickly and easily can the model be updated • Uptime/reliability 11
- 12. Model Governance Models should be: • governed by the company’s policies and procedures, laws and regulations and organization’s goals • be transparent, explainable, traceable and interpretable for auditors and regulators. • have approval and release process. • have reason code for explanation of decision. • be versioned. Reasoning for introduction of new version should be clearly specified. 12
- 13. What are we building We want to build a streaming system allowing to update models without interruption of execution (dynamically controlled stream). 13
- 14. Model Serving with Apache Flink Idea: Exploit Flink’s low latency capabilities for serving models. Focus on offline models loaded from a permanent storage and update them without interruption. FLIP Proposal: (https://docs.google.com/document/d/1ON_t9S28_2LJ91Fks2yFw0RYyeZvIvndu8 oGRPsPuk8) Combines different efforts: https://github.com/FlinkML ● https://github.com/FlinkML/flink-jpmml (https://radicalbit.io/) ● https://github.com/FlinkML/flink-modelServer (Boris Lublinsky) ● https://github.com/FlinkML/flink-tensorflow (Eron Wright) 14 14
- 15. Pros ● Live model updating ● No Flink runtime changes ● Latency guarantees ● Not all systems cover the requirements. For example TF Serving: ○ Metadata not available. (https://github.com/tensorflow/serving/issues/612) ○ No new models at runtime: (https://github.com/tensorflow/serving/issues/422) 15
- 16. Flink Low Level Join • Create a state object for one input (or both) • Update the state upon receiving elements from its input • Upon receiving elements from the other input, probe the state and produce the joined result 16
- 17. Exporting Model Different ML packages have different export capabilities. We used the following: • For Spark ML export to PMML - https://github.com/jpmml/jpmml-sparkml • For Tensorflow: • Normal graph export • Saved model format 17
- 18. Model representationOn the wire syntax = "proto3"; // Description of the trained model. message ModelDescriptor { // Model name string name = 1; // Human readable description. string description = 2; // Data type for which this model is applied. string dataType = 3; // Model type enum ModelType { TENSORFLOW = 0; TENSORFLOWSAVED = 2; PMML = 2; }; ModelType modeltype = 4; oneof MessageContent { // Byte array containing the model bytes data = 5; string location = 6; } } 18 Internal trait Model { def score(input : AnyVal) : AnyVal def cleanup() : Unit def toBytes() : Array[Byte] def getType : Long } trait ModelFactoryl { def create(input : ModelDescriptor) : Model def restore(bytes : Array[Byte]) : Model }
- 19. Key based join Flink’s CoProcessFunction allows key-based merge of 2 streams. When using this API, data is key-partitioned across multiple Flink executors. Records from both streams are routed (based on key) to the appropriate executor that is responsible for the actual processing. 19
- 20. Partition based join Flink’s RichCoFlatMapFunction allows merging of 2 streams in parallel (based on parallelization parameter). When using this API, on the partitioned stream, data from different partitions is processed by dedicated Flink executor. 20
- 21. Monitoring case class ModelToServeStats( name: String, // Model name description: String, // Model descriptor modelType: ModelDescriptor.ModelType, // Model type since : Long, // Start time of model usage var usage : Long = 0, // Number of servings var duration : Double = .0, // Time spent on serving var min : Long = Long.MaxValue, // Min serving time var max : Long = Long.MinValue // Max serving time ) Model monitoring should provide information about usage, behavior, performance and lifecycle of the deployed models 21
- 22. Queryable state Queryable state (interactive queries) is an approach, which allows to get more from streaming than just the processing of data. This feature allows to treat the stream processing layer as a lightweight embedded database and, more concretely, to directly query the current state of a stream processing application, without needing to materialize that state to external databases or external storage first. 22
- 23. Queryable state Limitations Operator state (non keyed) is not queryable https://issues.apache.org/jira/browse/FLINK-7771 23
- 24. Where are we now? • Proposed solution does not require modification of Flink proper. It rather represents a framework build leveraging Flink capabilities. Although extension of the queryable state will be helpful • Flink improvement proposal (Flip23) - model serving is close to completion - https://docs.google.com/document/d/1ON_t9S28_2LJ91Fks2yF w0RYyeZvIvndu8oGRPsPuk8). • Fully functioning prototype of solution supporting PMML and Tensorflow is currently in the FlinkML Git repo (https://github.com/FlinkML) 24
- 25. Next steps • Complete Flip23 document • Integrate Flink Tensorflow and Flink PMML implementations to provide PMML and Tensorflow specific model implementations - work in progress • Look at bringing in additional model implementations • Look at model governance and management implementation 25
- 26. Model Serving Beyond Flink Author: Boris Lublinsky Download from here: https://info.lightbend.com/ebook-serving-machine-learning-models-register.html 26