Splice Machine's use of Apache Spark and MLflow
1 like653 views
Splice Machine offers a scale-out RDBMS that supports both OLTP and OLAP for real-time AI applications, catering to sectors like financial services and healthcare. It integrates Apache Spark and MLflow to streamline data processing and machine learning workflows. Key features include a dual-engine architecture, deployment options across on-premise and cloud environments, and tools to address data science challenges.
Splice Machine's use of Apache Spark and MLflow
- 1. WIFI SSID:SparkAISummit | Password: UnifiedAnalytics
- 2. Gene Davis, Splice Machine Splice Machine’s use of Apache Spark and MLflow #UnifiedAnalytics #SparkAISummit
- 3. Splice Machine • What are we? – A scale-out RDBMS that enables simultaneous transactions (OLTP) and analytics (OLAP) – Powers Operational AI: the ability to run AI applications in real time • Who uses us? – Companies in financial services, healthcare, supply chain, etc. – One example: 7PB, 2B record updates/day, 2M queries/day with sub- second response time • How do we do it? – Transactional SQL engine on top of HBase and Spark • ”Dual engine” architecture – Many delivery options (on-premise, cloud service (AWS, Azure, bespoke cloud, etc.)) 3#UnifiedAnalytics #SparkAISummit
- 4. Operational AI 4#UnifiedAnalytics #SparkAISummit INTELLIGENT DECISIONS Operational Database • Scale-out • OLTP • Fast Enterprise Data Warehouse • In-Memory • OLAP • Massively Parallel ARTIFICIAL INTELLIGENCE BUSINESS INTELLIGENCEML Models • Notebooks • Algorithms • Model Workflow OPERATIONAL INTELLIGENCE Integrated data platform for real-time AI applications On Premise
- 5. The Three Dimensions of Intelligence 5#UnifiedAnalytics #SparkAISummit OLTPOLAP What has happened in the past that might impact you? What is happening right now? What will happen in the future? ML
- 6. The Three Dimensions of Intelligence 6#UnifiedAnalytics #SparkAISummit OLTPOLAP Key platforms are duct-taped together leading to High Infrastructure Costs • Latency in Decision Making • Isolation from Business Processes ML
- 7. Intelligent Action - Before 7#UnifiedAnalytics #SparkAISummit
- 8. Intelligent Action - After 8#UnifiedAnalytics #SparkAISummit
- 9. Data Science Pain Points 9#UnifiedAnalytics #SparkAISummit Data Scientist Data Engineer ● Is my data ready to go? ● Is it still relevant? ● Do my features still align? ● The ETL process changed again – now what? ● The Data Scientist requested a different level of granularity – how do I do that?
- 10. Data Science Pain Points 10#UnifiedAnalytics #SparkAISummit Data Scientist Data Engineer ● Is my data ready to go? ● Is it still relevant? ● Do my features still align? ● The ETL process changed again – now what? ● The Data Scientist requested a different level of granularity – how do I do that? ● What data did I use? ● What algorithms/parameters gave the best model? ● Why didn’t I get the same results? ● What libraries are used? ● What model version is deployed?
- 11. MLflow and ML Manager • Splice Machine chose MLflow – MLflow Tracking: Track experiment runs and parameters – MLflow Models: packaging model artifacts • Splice ML Manager – Machine Learning on the Splice Machine Stack – MLflow Tracking and Models – Includes UI to Deploy to Amazon SageMaker 11#UnifiedAnalytics #SparkAISummit
- 12. ML Manager Architecture 12#UnifiedAnalytics #SparkAISummit On Premises Splice Machine Data Platform Native Spark Data Source Deployment Automation
- 13. Native Spark Datasource • Efficient interface from the Splice relational tables into Spark DataFrames (and back again) • No serialization/deserialization • Examples: – interestingDf = spliceContext.df(“select * from interesting_table”) – spliceContext.insert(dfWithData,’table_name’) 13#UnifiedAnalytics #SparkAISummit
- 14. Accessing MLflow Capabilities • Start with Splice’s MLManager – manager = MLManager() – Convenience class on top of MLflow • API’s – manager.create_experiment() – manager.set_active_experiment() – manager.create_new_run() – manager.log_param() – manager.log_metric() – manager.log_spark_model() 14#UnifiedAnalytics #SparkAISummit
- 17. ML Manager • Beta Launched in March – MLflow v0.8 • Available at cloud.splicemachine.com • MLManager() API Open Source at: – https://github.com/splicemachine/pysplice – (subject to change per MLflow 1.0 API) 17#UnifiedAnalytics #SparkAISummit
- 19. DON’T FORGET TO RATE AND REVIEW THE SESSIONS SEARCH SPARK + AI SUMMIT The picture can't be displayed.
- 20. Many Disparate Tools 20#UnifiedAnalytics #SparkAISummit Data Sources OLTP - Oracle, Cassandra, Dynamo OLAP - Redshift, Snowflake, S3 Notebooks Apache Zeppelin Jupyter Data Manipulation Python Pandas Scikit Spark Machine Learning MLLib, R Experimenta tion Tracking MLflow Deployment Sagemaker AzureML
- 21. Insurance Claim Example 21#UnifiedAnalytics #SparkAISummit
- 22. Insurance Claim Example 22#UnifiedAnalytics #SparkAISummit