01-introduction.ppt the paper that you can unless you want to join me because everyone else has been on the paper

Chapter 1
Introduction to Big Data Analytics
HeDs5221 - Big Data
Analytics in Health
By Eshete Derb Emiru(PhD)
esheted2010@gmail.com

Chapter 1:Introduction to Big
Data Analytics
•What is Big Data?
•What makes Big Data different from other
related “buzzwords”?
•What are we going to cover in this course?

What’s Big Data?
Big Data Analytics 3

Google Translate

Big Data: Some Examples
• Topic detection and tracking
• Trend analysis
• Social network analysis
• PageRank
• Predictive analytics
• Many others: healthcare, natural resources,
education, public sector, insurance,
transportation, finance and crime detection,
…

Google News

Google Trends

What is Big Data?
• Big data is a term for data sets that are so
large or complex that traditional data
processing application software's are
inadequate to deal with them.
• Challenges include capture, storage,
analysis, data curation, search, sharing,
transfer, visualization, querying, updating
and information privacy. [source:
Wikipedia]

What is Big Data?
• “Big data is data whose scale, distribution,
diversity, and/or timeliness require the
use of new technical architectures and
analytics to enable insights that unlock
new sources of business value.”
• [source: C. Manyika, Big Data: The next
frontier for innovation, competition, and
productivity, McKinsey Global Institute,
2011]

Characteristics of Big Data
• META Group (now Gartner) defined data
growth challenges and opportunities as
being three-dimensional, i.e. increasing
volume, velocity, and variety [Doug
Laney, 2001]
– Volume: the quantity of generated and stored
data
– Velocity: the speed at which data is generated
and processed
– Variety: the type and nature of data

• “Big data is high volume, high velocity,
and/or high variety information assets
that require new forms of processing to
enable enhanced decision making, insight
discovery and process optimization.”
[Gartner, 2012]

The Four V’s of Big Data
[source: IBM]

• Five V’s:
– Volume: scale
• 2.5 EB per day (300 TB, Library of Congress)
– Velocity: streaming
• In 60 seconds: 350,000 tweets, 300 hours of YouTube video,
171 million emails, 350 GB sensor data from a jet engine
– Variety: different forms
• Structured, Semi-structured, Unstructured
– Veracity: uncertainty
• Quality or fidelity
– Value
• Higher veracity, lower processing time -> higher value

Data Structures
• Variety: different forms
– Structured: databases, spreadsheets, …
– Semi-structured: textual files such as Web
pages, XML, …
– Unstructured: text documents, images,
videos, …
• Data growth is increasingly unstructured
– Social media: Facebook, Twitter, …

Differences from traditional
data analysis
• Distinct requirements
– Combining of multiple unrelated datasets
– Processing of large amounts of unstructured data
– Harvesting of hidden information in a time-sensitive
manner
• Newer techniques that leverage computational
resources
• Interdisciplinary
– Mathematics, statistics, computer science, subject matter
expertise
• Benefits
– Optimization, predictions, fault or fraud detection,
improved decision making, discoveries

Data analysis vs. Data analytics
• “Analysis is the separation of a whole into its
component parts, and analytics is the method of
logical analysis.” [source: Merriam-Webster
dictionary]
• “Analysis is really a heuristic activity, where
scanning through all the data the analyst gains
some insight. “ [source: Quora.com]
• “Analytics is about applying a mechanical or
algorithmic process to derive the insights for
example running through various data sets
looking for meaningful correlations between
them. ” [source: Quora.com]

Related Terms
• Data science, predictive analytics
• Business intelligence, FinTech
• IoT, CPS, Industry 4.0
• Smart homes, smart cities
• Data mining, machine learning, artificial
intelligence
• Cloud computing, data-intensive computing,
parallel computing, distributed computing
• …

What is Data Science?

Data Engineering vs. Data
Analysis
• Data engineering: designing and building
infrastructure for integrating and managing data from
various resources
– MySQL, NoSQL, Hadoop, MapReduce
• Data analysis: querying and processing data,
providing reports, summarizing and visualizing data
– Statistics, visualization, Excel, SAS, SPSS, …
• Data science: applying statistics, machine learning and
analytic approaches to solve critical business
problems, and turning data into valuable and
actionable insights
– Advanced data analysis
– Data mining tools, machine learning, statistics, …

Big data vs. Business
Intelligence
• Business Intelligence (BI) are the set of
strategies, processes, applications, data,
products, technologies and technical
architectures which are used to support
the collection, analysis, presentation and
dissemination of business information.
[source: Wikipedia]

23
Data Mining in Business Intelligence
Increasing potential
to support
business decisions
End User
Business
Analyst
Data
Analyst
DBA
Decision
Making
Data Presentation
Visualization Techniques
Data Mining
Information Discovery
Data Exploration
Statistical Summary, Querying, and Reporting
Data Preprocessing/Integration, Data Warehouses
Data Sources
Paper, Files, Web documents, Scientific experiments, Database Systems
[source: Han 2011]
Big Data Analytics

Big Data vs. Financial
Technology
[source: Slideshare]

Big Data vs. IoT
• The Internet of things (IoT) is the
internetworking of physical devices,
vehicles (also referred to as "connected
devices" and "smart devices"), buildings,
and other items—embedded with
electronics, software, sensors, actuators,
and network connectivity that enable
these objects to collect and exchange data.
[source: Wikipedia]

Big Data vs. Industry 4.0
[source: Roland Berger]

Big Data vs. CPS
• A cyber-physical system (CPS) is a mechanism
controlled or monitored by computer-based
algorithms, tightly integrated with the internet and its
users.
• In cyber-physical systems, physical and software
components are deeply intertwined, each operating on
different spatial and temporal scales, exhibiting
multiple and distinct behavioral modalities, and
interacting with each other in a myriad of ways that
change with context.
– E.g.: smart grid, autonomous automobile systems, medical
monitoring, process control systems, robotics systems, and
automatic pilot avionics
– [source: Wikipedia]

Big Data vs. Data Mining
• Data mining is the computational process
of discovering patterns in large data sets
involving methods at the intersection of
artificial intelligence, machine learning,
statistics, and database systems. [source:
Wikipedia]

29
Knowledge Discovery (KDD) Process
• This is a view from typical
database systems and data
warehousing communities
• Data mining plays an essential
role in the knowledge discovery
process
Data Cleaning
Data Integration
Databases
Data
Warehouse
Task-relevant Data
Selection
Data Mining
Pattern Evaluation
[source: Han 2011]
Big Data Analytics

Big Data vs. Data-Intensive
Computing
• Data-intensive computing is a class of
parallel computing applications which use
a data parallel approach to process large
volumes of data typically terabytes or
petabytes in size and typically referred to
as big data. [source: Wikipedia]

Google Trends Comparison:
Big data, IoT, Cloud computing

Big data vs. AI, ML, and DL

Limits of Predictions
• Predictive analytics: technology that learns
from experience (data) to predict the future
behavior of individuals in order to drive
better decisions
• Accurate prediction is generally not possible
• But predictions need not be accurate to bring
value
– E.g. direct mail marketing
• The prediction effect: predicting better than
pure guess delivers value

Is Big Data the End of Theory?
• Critiques
– Big data requires “big judgement”
– If the systems dynamics of the future change,
the past can say little about the future
– Privacy
– Bias, subjective, shallow
– “If you believe in Big Data analytics, it’s time
to begin planning for a Hillary Clinton
presidency and all that entails.”

What are we going to cover in
this course?
• Data mining
– Frequent pattern mining
– Classification
– Clustering
• Parallel programming in distributed
platforms
– Scalability
– Hadoop, Spark
– MapReduce programming

36
KDD Process: A Typical View from ML and Statistics
Input Data Data
Mining
Data Pre-
Processing
Post-
Processin
g
• This is a view from typical machine learning and statistics communities
Data integration
Normalization
Feature selection
Dimension reduction
Pattern discovery
Association &
correlation
Classification
Clustering
Outlier analysis
… … … …
Pattern evaluation
Pattern selection
Pattern interpretation
Pattern visualization
[source: Han 2011
Big Data Analytics

37
Data Mining Function: Association
and Correlation Analysis
• Frequent patterns (or frequent itemsets)
– What items are frequently purchased together in
your Walmart?
• Association, correlation vs. causality
– A typical association rule
• Diaper  Beer [0.5%, 75%] (support, confidence)
– Are strongly associated items also strongly
correlated?
• How to mine such patterns and rules efficiently in
large datasets?
• How to use such patterns for classification,
clustering, and other applications?
Big Data Analytics

38
Data Mining Function:
Classification
• Classification and label prediction
– Construct models (functions) based on some training examples
– Describe and distinguish classes or concepts for future prediction
• E.g., classify countries based on (climate), or classify cars based on
(gas mileage)
– Predict some unknown class labels
• Typical methods
– Decision trees, naïve Bayesian classification, support vector
machines, neural networks, rule-based classification,
pattern-based classification, logistic regression, …
• Typical applications:
– Credit card fraud detection, direct marketing,
classifying stars, diseases, web-pages, …
Big Data Analytics

39
Data Mining Function: Cluster
Analysis
• Unsupervised learning (i.e., Class label is
unknown)
• Group data to form new categories (i.e.,
clusters), e.g., cluster houses to find
distribution patterns
• Principle: Maximizing intra-class similarity
& minimizing interclass similarity
• Many methods and applications
Big Data Analytics

Limitations of Data Analysis
• CPU: computing time to execute the
analysis
• I/O: how much data can be put in
memory per time unit
• Memory: how much data can be processed
at a time

Data to be analyzed
• Tall data: large number of cases
• Wide data: large number of features
• Tall and wide data: large number of both
cases and features
• Sparse data: large number of zero entries

Algorithm to be used
• How complex is your algorithm
• How many parameters in your model
• Are the optimization processes
parallelizable
• Does your algorithm learn from all data or
small batches of data

Possible solutions
• Scale up: performance improvement on a single
machine
– More memory, faster CPU, faster storage, using
GPUs, …
– E.g. CUDA, TensorFlow, Keras, …
• Scale out: performance improvement by
distributing computations
– Using outside resources: other CPUs, GPUs, storage
– E.g. Hadoop, Spark, …
• Scale up and out
– E.g. Distributed TensorFlow, …

Hadoop Architecture

MapReduce

Functional Programming

Spark and Hadoop

TensorFlow
• An open-source library for machine
learning
– High-level API: Keras, …
– Low-level API: TensorFlow Core
• Prerequisites: algebra, Python
• Google Colab (Colaboratory) is an easy
way to learn and use TensorFlow

TensorFlow

Typical Workflows in
TensorFlow
Input data Tensors
Model
Model training
and testing
Dataflow Graph
device device
sessions

Thanks for Your Attention!

01-introduction.ppt the paper that you can unless you want to join me because everyone else has been on the paper

More Related Content

Similar to 01-introduction.ppt the paper that you can unless you want to join me because everyone else has been on the paper

Recently uploaded

01-introduction.ppt the paper that you can unless you want to join me because everyone else has been on the paper