Big Data Modeling Challenges and Machine Learning with No Code

designed by tinyPPT.com
Karthik Chinnusamy
Sr Principal,Data Architect
Veritas Tech
SFBayACM Presentation
Wednesday, January 15, 2020
7:00 PM to 9:00 PM
Palo Alto Networks Building 3
3200 Tannery Way · Santa Clara, ca
Big Data
Modeling and
Machine
Learning with
no code

BIG DATA
INTRODUCTION
 Machine
 People
 Organization
SOURCES
 Integrating different types of
data sources
Reduce Complexity
Increase data availability
Data collaboration
VALUE
 Volume
 Velocity
 Veracity
CHARACTERSTICS
 Big data
 Insight
 Action => Product
DATA SCIENCE
DATA MANAGEMENT
DATA
GOVERNANCE
 Variety
 Valence

BIG DATA
PROCESSING
 Identify Data
 Retrieve Data
 Query Data
ACQUIRE
 Explore
 Pre-process
PREPARE
 Select analytical
techniques
 Build models
ANALYZE
 Communicate
results
REPORT
 Apply results
ACT

ACQUIRE
PREPARE
ANALYZEREPORT
ACT
Iterative
Process
Enable
Scalability
Provide a value
Facilitate share
environment
Optimized for
variety of data
Handle fault
tolerance Scalable
Tools

Big data eco-system

 Create a nice
 Easy-to-navigate
 Easy-to-understand
 Easy-to-maintain
 Easy-to-document
 Easy-to-consume architectural picture of these application and data chaos?
 How do they do so when faced with time pressure to comply with demanding
regulations like GDPR, BCBS 239, CMS, and others?
 What’s the starting point?
 Do you start at the end with the reports?
 But which reports first?
Data Governance & Information Architect Challenges
 Where can I find them?
 Who owns them?
 Who maintains them?
 Who can help me explain them?
 Are they still up-to-date?

Machine Learning
with KNIME
KNIME Open source
Analytic Platform
Server
GUI Based
No Coding
Experience required
Extensions
& Community
Extensions
ADVANCED
VISUALIZATION
Extensive
Integration
K1

K1 Karthik-ASUS, 9/11/2018

KNIME
• KNIME founded in 2008
• KoNstanz Information MinEr(KNIME)
• KNIME AG located in Zurich and the group of Michael Berthold at the
University of Konstanz, Chair for Bioinformatics and Information Mining.
• Interactive Preprocessing, Analysis and Modelling
• Repeatable and reproduceable
• Integration with Python,Java, R &SPARK
• Integration with AWS and AZURE
• Integration with Weka,Plotly,Keras,H2O
• Waikato Environment for Knowledge Analysis, developed at the University of
Waikato, New Zealand
• Plotly is a technical computing company headquartered in Montreal, Quebec,
that develops online data analytics and visualization tools.
• Keras is an open-source neural-network library written in Python. Run top of
TensorFlow, Microsoft Cognitive Toolkit, R, Theano, or PlaidML.
• H2O, the leading open source data science and machine learning platform
trusted by over 18,000 organizations and 200,000 users around the world.

KNIME Overview

KNIME Nodes and
Status

Orchestration & Workflow Oozie, ODE, Airavata and OODT (Tools)
NA: Pegasus, Kepler, Swift, Taverna, Trident, ActiveBPEL, BioKepler, Galaxy
Data Analytics Libraries:
Machine Learning
Mahout , MLlib , MLbase
CompLearn (NA)
Linear Algebra
Scalapack, PetSc (NA)
Statistics, Bioinformatics
R, Bioconductor (NA)
Imagery
ImageJ (NA)
MRQL
(SQL on Hadoop,
Hama, Spark)
Hive
(SQL on
Hadoop)
Pig
(Procedural
Language)
Shark
(SQL on
Spark, NA)
Hcatalog
Interfaces
Impala (NA)
Cloudera
(SQL on Hbase)
Swazall
(Log Files
Google NA)
High Level (Integrated) Systems for Data Processing
Parallel Horizontally Scalable Data Processing
Giraph
~Pregel
Tez
(DAG)
Spark
(Iterative
MR)
Storm
S4
Yahoo
Samza
LinkedIn
Hama
(BSP)
Hadoop
(Map
Reduce)
Pegasus
on Hadoop
(NA)
NA:Twister
Stratosphere
Iterative MR
GraphBatch Stream
Pub/Sub Messaging Netty (NA)/ZeroMQ (NA)/ActiveMQ/Qpid/Kafka
ABDS Inter-process Communication
Hadoop, Spark Communications MPI (NA)
& Reductions Harp Collectives (NA)
HPC Inter-process Communication
Cross Cutting
Capabilities
DistributedCoordination:ZooKeeper,JGroups
MessageProtocols:Thrift,Protobuf(NA)
Security&Privacy
Monitoring:Ambari,Ganglia,Nagios,Inca(NA)

In memory distributed databases/caches: GORA (general object from NoSQL), Memcached
(NA), Redis(NA) (key value), Hazelcast (NA), Ehcache (NA);
Mesos, Yarn, Helix, Llama(Cloudera) Condor, Moab, Slurm, Torque(NA) ……..
ABDS Cluster Resource Management HPC Cluster Resource Management
ABDS File Systems User Level HPC File Systems (NA)
HDFS, Swift, Ceph FUSE(NA) Gluster, Lustre, GPFS, GFFS
Object Stores POSIX Interface Distributed, Parallel, Federated
iRODS(NA)
Interoperability Layer Whirr / JClouds OCCI CDMI (NA)
DevOps/Cloud Deployment Puppet/Chef/Boto/CloudMesh(NA)
Cross Cutting
Capabilities
DistributedCoordination:ZooKeeper,JGroups
MessageProtocols:Thrift,Protobuf(NA)
Security&Privacy
Monitoring:Ambari,Ganglia,Nagios,Inca(NA)
SQL
MySQL
(NA)
SciDB
(NA)
Arrays,
R,Python
Phoenix
(SQL on
HBase)
UIMA
(Entities)
(Watson)
Tika
(Content)
Extraction Tools
Cassandra
(DHT)
NoSQL: Column
HBase
(Data on
HDFS)
Accumulo
(Data on
HDFS)
Solandra
(Solr+
Cassandra)
+Document
Azure
Table
NoSQL: Document
MongoDB
(NA)
CouchDB Lucene
Solr
Riak
~Dynamo
NoSQL: Key Value (all NA)
Dynamo
Amazon
Voldemort
~Dynamo
Berkeley
DB
Neo4J
Java Gnu
(NA)
NoSQL: General Graph
RYA RDF on
Accumulo
NoSQL: TripleStore RDF SparkQL
AllegroGraph
Commercial
Sesame
(NA)
Yarcdata
Commercial
(NA)
Jena
ORM Object Relational Mapping: Hibernate(NA), OpenJPA and JDBC Standard
File
Management
IaaS System Manager Open Source Commercial Clouds
OpenStack, OpenNebula, Eucalyptus, CloudStack, vCloud, Amazon, Azure, Google
Bare
Metal
Data Transport BitTorrent, HTTP, FTP, SSH Globus Online (GridFTP)

KNIME
•Open source KNIME Analytics Platform for creating data
science
•Commercial KNIME Server for productionizing data
science.
KNIME Analytics Platform is the open source software for
creating data science. Intuitive, open, and continuously
integrating new developments, KNIME makes
understanding data and designing data science
workflows and reusable components accessible to
everyone.
KNIME Server is the enterprise software for team-based
collaboration, automation, management, and deployment
of data science workflows as analytical applications and
services. Non experts are given access to data science
via KNIME WebPortal or can use REST APIs
Extensions for KNIME Analytics Platform

Demo
KNIME Software in the Cloud
KNIME Analytics Platform and KNIME Server are
available on Microsoft Azure and Amazon AWS

KNIME Workbench

Business Usecase

Acknowledgement &
Additional Info
• www.knime.com
• https://www.knime.com/resources

Appendix

KNIME Cheat Sheet

Big Data Modeling Challenges and Machine Learning with No Code

More Related Content

What's hot (19)

Similar to Big Data Modeling Challenges and Machine Learning with No Code (20)

Recently uploaded (20)

Big Data Modeling Challenges and Machine Learning with No Code