Heterogeneous Data Mining with Spark
0 likes700 views
The document provides an overview of KNIME, an analytics platform that supports various data mining techniques and languages, including integration with Apache Spark. It highlights KNIME's capabilities in handling heterogeneous data sources, transforming and visualizing data, and detailed applications in sectors like healthcare and finance. The document also discusses the challenges of analyzing unstructured data and the tools available in KNIME to address these challenges.
Heterogeneous Data Mining with Spark
- 1. © 2017 KNIME.com AG. All Rights Reserved. Heterogeneous Data Mining with Spark Tobias Kötter KNIME
- 2. 2© 2017 KNIME.com AG. All Rights Reserved. What is KNIME?
- 3. © 2017 KNIME.com AG. All Rights Reserved. 3 The KNIME® Analytics Platform
- 4. © 2017 KNIME.com AG. All Rights Reserved. 4 Analysis & Mining Statistics, Machine Learning, Data Mining, Web Analytics, Text Mining, Network Analysis, Social Media Analysis, R, Weka, Python, Community / 3rd party, ... Data Access MySQL, Oracle, ... SAS, SPSS, ... Excel, Flat, ... Hive, Impala, ... XML, JSON, PMML Text, Doc, Image, ... Web Crawlers, Industry Specific, Community / 3rd party ... Transformation Row, Column, Matrix Text, Image, Networks, Time Series, Java, Python, Community / 3rd party, ... Visualization R, Python, JFreeChart, JavaScript, Community / 3rd party, ... Deployment via BIRT PMML, XML, JSON Databases, Excel, Flat, etc. Text, Doc, Image Industry Specific Community / 3rd party, ... Over 1500 native and embedded nodes included: Big Data Hive, Impala, HDFS Vertica, Teradata/Aster, Spark, MLlib, Community / 3rd party, ...
- 5. © 2017 KNIME.com AG. All Rights Reserved. 5 Broad Range of KNIME Application Areas & Customers Advanced Analytics Pharma Health Care Finance Retail Customer Intelligence Manu- facturing
- 6. © 2017 KNIME.com AG. All Rights Reserved. 6 KNIME Analytics Platform: Try it Now! • Download from www.knime.com • Browse the KNIME Learning Hub at www.knime.com/learning-hub • Check out KNIME Press “KNIME Beginner’s Guide” • Become active on our Forum!
- 7. © 2017 KNIME.com AG. All Rights Reserved. 7 KNIME Software Overview
- 8. © 2017 KNIME.com AG. All Rights Reserved. 8 KNIME Big Data Connectors • Package required drivers/libraries for HDFS, Hive, Impala access • Runs on Hadoop • Preconfigured connectors – Hive – Cloudera Impala – (secured) HDFS, webHDFS, httpFS • Support for Kerberos secured cluster • Extends the open source database and remote file handling integration
- 9. © 2017 KNIME.com AG. All Rights Reserved. 9 Database Extension • Visually assemble complex SQL statements • Connect to almost all JDBC-compliant databases • Harness the power of your database within KNIME • Operations are performed within the database
- 10. © 2017 KNIME.com AG. All Rights Reserved. 10 KNIME Spark Executor • Commercial extension • Based on Spark MLlib • Scalable machine learning library • Algorithms for – Classification (decision tree, naïve bayes, …) – Regression (logistic regression, linear regression, …) – Clustering (k-means) – Collaborative filtering (ALS) – Dimensionality reduction (SVD, PCA) • Supports Spark version 1.2, 1.3, 1.5, 1.6, 2.0, 2.1 and 2.2 • Kerberos secured cluster support
- 11. © 2017 KNIME.com AG. All Rights Reserved. 11 Machine Learning – Supervised Learning Example
- 12. © 2017 KNIME.com AG. All Rights Reserved. 12 Let KNIME Control Your Spark Jobs
- 13. 13© 2017 KNIME.com AG. All Rights Reserved. Use Case
- 14. © 2017 KNIME.com AG. All Rights Reserved. 14 The Question • Wouldn’t it be great to know if your flight will be delayed? https://pixabay.com/en/staircase-airport-modern-technology-1149599/
- 15. © 2017 KNIME.com AG. All Rights Reserved. 15 The Answer • Of course, so let’s learn a model that does! https://pixabay.com/en/banner-yes-no-decision-choice-1183407/
- 16. © 2017 KNIME.com AG. All Rights Reserved. 16 The Airport • Chicago O’Hare International Airport https://commons.wikimedia.org/wiki/File:O%27Hare_with_AA_plane.JPG Foto Ad Meskens
- 17. © 2017 KNIME.com AG. All Rights Reserved. 17 The Airport • Flughafen Berlin Brandenburg https://de.wikipedia.org/wiki/Datei:BBI_2010-07-23_5.JPG
- 18. © 2017 KNIME.com AG. All Rights Reserved. 18 The Data • Historical Flight Data • Airport and City Information • Geo Coordinates • Airplane Data • Radar Images • Textual Weather Reports https://commons.wikimedia.org/wiki/File:World-airline-routemap-2009.png
- 19. © 2017 KNIME.com AG. All Rights Reserved. 19 The Challenges • Many Data Sources and Formats • Large Unstructured Data • Analyze the Data
- 20. © 2017 KNIME.com AG. All Rights Reserved. 20 The Challenges • Many Data Sources and Formats • Large Unstructured Data • Analyze the Data
- 21. © 2017 KNIME.com AG. All Rights Reserved. 21 Spark Reader and Writer Nodes • Read and write various data formats from scalable storage e.g. HDFS • Data preview in the node dialog • Based on the Spark Data Source API
- 22. © 2017 KNIME.com AG. All Rights Reserved. 22 CSV to Spark Node
- 23. © 2017 KNIME.com AG. All Rights Reserved. 23 CSV to Spark Node
- 24. © 2017 KNIME.com AG. All Rights Reserved. 24 Virtual Data Warehouse
- 25. © 2017 KNIME.com AG. All Rights Reserved. 25 The Challenges • Many Data Sources and Formats • Large Unstructured Data • Analyze the Data https://pixabay.com/en/files-paper-office-paperwork-stack-1614223/
- 26. © 2017 KNIME.com AG. All Rights Reserved. 26 Radar Data Analysis
- 27. © 2017 KNIME.com AG. All Rights Reserved. 27 Radar Images
- 28. © 2017 KNIME.com AG. All Rights Reserved. 28 Image Processing Result
- 29. © 2017 KNIME.com AG. All Rights Reserved. 29 KNIME Image Processing on Spark
- 30. © 2017 KNIME.com AG. All Rights Reserved. 30 KNIME Image Processing on Spark
- 31. © 2017 KNIME.com AG. All Rights Reserved. 31 Sentiment Analysis https://pixabay.com/en/emotions-man-happy-sad-face-adult-371238/
- 32. © 2017 KNIME.com AG. All Rights Reserved. 32 Textual Weather Reports
- 33. © 2017 KNIME.com AG. All Rights Reserved. 33 Text Processing Result
- 34. © 2017 KNIME.com AG. All Rights Reserved. 34 KNIME Text Processing on Spark
- 35. © 2017 KNIME.com AG. All Rights Reserved. 35 KNIME Text Processing on Spark
- 36. © 2017 KNIME.com AG. All Rights Reserved. 36 The Challenges • Many Data Sources and Formats • Large Unstructured Data • Analyze the Data https://pixabay.com/en/ball-binary-magnifying-glass-hand-958950/
- 37. © 2017 KNIME.com AG. All Rights Reserved. 37 Ad-hoc Analysis and Model Learning on Spark
- 38. © 2017 KNIME.com AG. All Rights Reserved. 38 Model Learning on Spark
- 39. © 2017 KNIME.com AG. All Rights Reserved. 39 Decision Tree Model
- 40. © 2017 KNIME.com AG. All Rights Reserved. 40 Ad-hoc Analysis on Spark
- 41. © 2017 KNIME.com AG. All Rights Reserved. 41 Do you speak SQL? • Spark SQL Query node with syntax highlighting and query completion
- 42. © 2017 KNIME.com AG. All Rights Reserved. 42 OSM Map View with Delay by Destination
- 43. © 2017 KNIME.com AG. All Rights Reserved. 43 Bar Chart with Delay by Destination
- 44. 44© 2017 KNIME.com AG. All Rights Reserved. Behind the Scene
- 45. © 2017 KNIME.com AG. All Rights Reserved. 45 Behind the Scene Cluster Worker NodeCluster Worker Node KNIME Workflow KNIME Analytics Platform KNIME Server Workflow Replica Execute KNIME workflow on Spark RDD Partition RDD Partition Input RDD RDD Partition RDD Partition Output RDD Spark Executor JVMSpark Executor JVM KNIME Workflow (OSGI) KNIME Workflow (OSGI)
- 46. © 2017 KNIME.com AG. All Rights Reserved. 46 Behind the Scene Cluster Worker NodeCluster Worker Node KNIME Workflow KNIME Analytics Platform KNIME Server Workflow Replica Execute KNIME workflow on Spark RDD Partition RDD Partition Input RDD RDD Partition Output RDD Spark Executor JVMSpark Executor JVM KNIME Workflow (OSGI) • Variation (1): Send RDD data through a single workflow replica RDD Partition
- 47. © 2017 KNIME.com AG. All Rights Reserved. 47 Behind the Scene Cluster Worker NodeCluster Worker Node KNIME Workflow KNIME Analytics Platform KNIME Server Workflow Replica Execute KNIME workflow on Spark RDD Partition Input RDD Spark Executor JVMSpark Executor JVM KNIME Workflow (OSGI) • Variation (2): Send pre- grouped RDD data through workflow replicas RDD Partition KNIME Workflow (OSGI) RDD Partition Output RDD RDD Partition RDD Partition RDD Partition
- 48. © 2017 KNIME.com AG. All Rights Reserved. 48 Summary • Visual assembling of Spark jobs – No coding required – Works together with other KNIME nodes e.g. loops • Spark Data Source API to read from various sources – Supports various data formats – Read form any JDBC compliant database • Prototype to use “all” KNIME nodes in Spark
- 49. 49© 2017 KNIME.com AG. All Rights Reserved. The KNIME® trademark and logo and OPEN FOR INNOVATION® trademark are used by KNIME.com AG under license from KNIME GmbH, and are registered in the United States. KNIME® is also registered in Germany.