Sparkflows Use Cases
Download as PPTX, PDF•0 likes•295 views
This document provides an overview of the Sparkflows solution for building and deploying self-serve big data analytics applications and use cases in under 30 minutes. It highlights key features such as over 100 building blocks for ETL, machine learning, NLP, OCR and connecting to various data sources/sinks. Example use cases demonstrated in under 30 minutes include building ETL pipelines, performing NLP and OCR on big data, streaming analytics, machine learning, entity resolution, log analytics, format conversion and loading data into systems like Solr, Elastic Search and HBase. It also covers creating custom nodes and dashboards.
Sparkflows Use Cases
- 1. Use Cases to Build & Deploy in < 30 min Self-Serve Big Data Analytics & Applications
- 3. 3 100 + Building Blocks ETL, ML, OCR, NLP, Connect to various Sources/Sinks Workflow Editor Powerful Schema Inference, Schema Propagation, Interactive Execution Visualization & DashboardsPrebuilt Workflows Introduction
- 4. 4 Workflow Editor Sparkflows Solution Rich Visualizations & Dashboards 100’s of Pre- built Nodes Batch & Streaming Engine Interactive Execution Easy Deployment & Configuration Pre-built Workflows Telco Churn Pred Housing Price Pred Bike Sharing Analysis NY Taxi Data Analysis Movie Lens Recommendations
- 5. 5 Sparkflows Product Stack Streaming Data Kafka Flume Data Sources HIVE/HBase HDFS/S3 Solr RDBMS Apache Spark Cluster Databricks AWS IBM Bluemix On Prem Azur e Data Sinks HIVE/HBase HDFS/S3 Solr RDBMS Visualizations / Dashboards
- 6. 6 Machine Learning Classification Regression Clustering Collaborative Filtering Save/Load Model Predict Cross-Validator NLP NER Sentiment OCR Tesseract Visualization Line Chart Bar Chart Pie Chart Updating Dashboards File Formats CSV/TSV Parquet JSON Avro PDF Images Whole Files Feature Generation Tokenization TF, IDF OneHotEncoder StringIndexer Imputer Scaler Data Sources/Sinks HDFS S3 Kafka, Flume, Twitter HBase Solr Elastic Search ETL Joins, Unions Filter SQL, Scala, Python GeoIP ConcatColumns Column Filter Dedup Languages SQL Scala Jython Java Some of the Building Block / Nodes
- 7. 7 Use Cases in < 30 minutes Self-Serve Big Data Analytics ETL Pipelines NLP OCR Streaming Analytics Do Big Data Analytics with Drag & Drop with 100+ building blocks Build ETL pipelines with ease. Also incorporate SQL, Scala, Jython in it. Perform NLP on Big Data with OpenNLP and Stanford CoreNLP Perform OCR on millions of images with Tesseract Perform Streaming Analytics reading from Kafka, performing complex transforms, generate graphs and write out to Solr, Hbase etc.
- 8. 8 Use Cases in < 30 minutes Machine Learning Entity Resolution Log Analytics Format Conversion Load data into Solr, ES, HBase Perform Machine Learning on huge datasets with drag and drop Perform large scale Entity Resolution on data from multiple channels Build Log Analytics Platform with Kafka, Spark, Solr/Elastic Search, Hue Convert Big Data from one format to another Easily load data into Solr, Elastic Search, HBase etc.
- 9. 9 Use Cases in < 30 minutes Custom Nodes Create Custom Nodes and drop them in the Library/Workflow Editor Dashboards Combine various outputs of workflows into a Dashboard
- 10. Self-Serve Data Analytics Spark CSV Read AVRO Save JSON Parquet Solr HBase Elastic Search HIVE Row Filter / Rename Col Random Forest SQL / Scala / Jython JOIN Read Graph Graph Model Dashboard
- 11. ETL – Build ETL pipelines with ease HIVE Solr Spark CSV Filter Filter JOIN SQL ES HBase HIVE LoadSolr LoadES LoadHBase LoadHIVE ReadCSV ReadHIVE
- 12. ETL – Connect various SQL for powerful pipelines HIVE Solr Spark CSV SQL SQL SQL SQL ES HBase HIVE LoadSolr LoadES LoadHBase LoadHIVE ReadCSV ReadHIVE
- 13. NLP – Perform distributed NLP on Big Data CSV Solr Spark PDF NLP NLP JOIN ES HBase HIVE LoadSolr LoadES LoadHBase LoadHIVE ReadPDF ReadCSV
- 14. OCR – Perform distributed OCR on Big Data Solr Spark PDF OCR ES HBase HIVE LoadSolr LoadES LoadHBase LoadHIVE ReadPDF Plus extract images
- 15. Streaming Analytics – With Kafka & Spark Streaming Solr Spark ES HBase HIVE LoadSolr LoadES LoadHBase LoadHIVE ReadKafka Apply various transforms K a f k a Transform Graph
- 16. Machine Learning – With Spark ML Spark Logistic Regression Score Evaluate Apply various transforms TransformHIVE Split
- 17. Entity Resolution – Applying various distance algorithms & scoring Spark Dedup Join & Transform DataSet 1 DataSet 2 HIVE Filter low Scores
- 21. Loading Data into Solr, Elastic Search, HBase, HIVE Spark CSV Read AVRO Save JSON Parquet Solr HBase Elastic Search HIVE
- 22. Custom Nodes – Create & Use Custom Nodes which add custom features Spark Custom Node Join & Transform DataSet 1 DataSet 2 HIVECustom Node
- 23. Dashboards – Combine output of various Workflows/Nodes into a Dashboard
- 24. 24 THANK YOU
Editor's Notes
- #2: Makes building Big Data Applications Agile, much, much faster and predictable
- #6: Benefits: Business Users Can Really Interact with Data & Experiment with Building Applications Rich Dashboards - Make day-to-day operations more efficient and provide insights into data and workflow performance Pre-Built Applications which can be easily extended or changed Use Cases Easy to Visualize and Implement