Big data analytics, survey r.nabati

Big Data Analytics
CE Dep, IAUSDJ.ac.ir
23-10-95(12 Jan, 2017)
Survey

 explosive increase of global data
 In 2011 overall created and copied data
volume in the world was 1.8ZB
 the term of big data is mainly used to
describe enormous datasets
 many government agencies announced
 major plans to accelerate big data research
and applications

 Big data is an abstract concept
 Different with “massive data” or “very big
data.”
 In 2010, Apache Hadoop defined big data as
“datasets which could not be captured,
managed, and processed by general
computers within an acceptable scope.”

 in May 2011, McKinsey & Company. Big data
shall mean such datasets which could not be
acquired, stored, and managed by classic
database software

 Doug Laney, an analyst of META (presently
Gartner)defined challenges and opportunities
brought about by increased data with a 3Vs
model
 The increase ofVolume,Velocity, andVariety

 The increase ofVolume,Velocity,Variety and
Big DataValue

 retailers that fully utilize big data may
improve their profit by more than 60 %
 developed economies in Europe could save
over EUR 100 billion (which excludes the
effect of reduced frauds, errors, and tax
difference)
 Extract wisdom from data

Data representation
 Data representation aims to make data more
meaningful for computer analysis and user
interpretation
 Levels of heterogeneity in type, structure,
semantics, organization, granularity, and
accessibility.

Redundancy reduction and data
compression
 There is high level of redundancy in datasets
 Is effective to reduce the indirect cost of the
entire system

Data life cycle management
 Values hidden in big data depend on data
freshness
 Which data shall be stored and which data
shall be discarded.

Analytical mechanism
 Traditional RDBMSs are strictly designed
with a lack of scalability and expandability
 We shall find a compromising solution
between RDBMSs and non-relational
databases.

Data confidentiality
 Distributed nodes and servers
 Third party for processing

Energy management
 From both economy and environment
perspectives
 With increase of data volume and analytical
demands, processing, storage, and
transmission of big data consume more
electric energy so:
 System-level power consumption control
 Management mechanism.

Expendability and scalability
 The analytical system of big data must
support present and future datasets.
 The analytical algorithm must be able to
process increasingly expanding and more
complex datasets.

Cooperation
 Analysis of big data is an interdisciplinary
research, which requires experts in different
fields cooperate.

 The main objective of cloud computing is to
use huge computing and storage resources
under concentrated management.
 The development of cloud computing
provides solutions for the storage and
processing of big data.
 The distributed storage technology based on cloud computing can
effectively manage big data
 The parallel computing capacity by virtue of cloud computing can
improve the efficiency of acquisition and analyzing big data.

 Cloud computing transforms the IT
architecture
 While big data influences business decision
making.
 Big data and cloud computing have different
 target customers
 Cloud computing is an advanced IT solution
 Big data focusing on business operations.

 Enormous amount of networking sensors are
embedded into various devices and machines
in the real world.
 collect various kinds of data, such as
environmental data, geographical data,
astronomical data, and logistic data.
 Mobile equipment's, transportation facilities,
public facilities, and home appliances could
all be data acquisition equipments in IoT

 Not only is a platform for concentrated
storage of data.
 acquiring data
 managing data
 organizing data
 leveraging the data values and functions.

 Hadoop is widely used in big data
applications in the industry, e.g., spam
filtering, network searching, clickstream
analysis, and social recommendation.
 Academic research is now based on Hadoop.
 In June 2012,Yahoo runs Hadoop in 42,000
servers at four data centers to support its
products and services, e.g., searching and
spam filtering

 Facebook announced that their Hadoop
cluster can process 100 PB data, which grew
by 0.5 PB per day as in November 2012.
 many companies provide Hadoop ommercial
execution and/or support, including Cloudera,
IBM, MapR, EMC, and Oracle.

Enterprise data
 Amazon processes millions of terminal
operations and more than 500,000 queries
from third-party sellers per day.
 Walmart processes one million customer
trades per hour, over 2.5PB.
 Akamai analyzes 75 million events per day

 Come from industry, agriculture, traffic,
transportation, medical care, public
departments, and families, etc.
 The data generated from IoT has the following
features:
 Large-scale data
 Heterogeneity
 Strong time and space correlation
 Effective data accounts for only a small portion of
the big data

 High-throughput bio-measurement
technologies are developed
 The completion of HGP (Human Genome
Project)
 One sequencing of human gene may
generate 100-600GB raw data.
 1.15 million human samples and 150,000
animal, plant, and microorganism samples in
China National Genebank, 2015, 30 million.

 Clinical medical care and medical R & D Data
 Pittsburgh Medical Center has stored 2TB such
data.
 By 2015, the average data volume of every
hospital will increase from 167TB to 665TB.

 Astronomy data, recorded 25TB data from
1998 to 2008.
 In the beginning of 2008, the Atlas
experiment of Large Hadron Collider (LHC) of
European Organization for Nuclear Research
generates raw data at 2PB/s and stores about
10TB processed data per year.
 Mobile data.
 Etc.

 Data collection
 Data transmission
 Data pre-processing

 Log files
 Sensing
 Methods for acquiring network data(crowlers
etc.)
 Lipcap-based pocket capture-easy to use library
 Zero copy packet capture
 Monitoring software’s:Wireshark, Smartsniff,
WinNetCap
 Mobile equipment’s

 Inter-DCN transmissions
 from data source to data center
 Intra-DCNTransmissions
 data communication flows within data centers
 High speed infrastructure

The collected datasets vary with respect to noise,
redundancy, and consistency, and it is undoubtedly a
waste to store meaningless data.
Integration
 Combination of data from different sources and
provides users with a uniform view of data
Cleaning
 Cleaning: data cleaning is a process to identify
inaccurate, incomplete, or unreasonable data,
Redundancy elimination

 Direct Attached Storage (DAS)
 Various hard disks are directly connected with
servers
 Network Storage
 Network Attached Storage (NAS)
 Storage Area Network (SAN).

 Consistency
 A distributed storage system requires multiple
servers to cooperatively store data
 Availability
 It would be desirable if the entire system is not
seriously affected to satisfy customer’s requests in
terms of reading and writing.
 PartitionTolerance
 The distributed storage still works well when the
network is partitioned

 Eric Brewer proposed a CAP theory in 2000,
which indicated that a distributed system
could not simultaneously meet the
requirements on consistency, availability, and
partition tolerance; at most two of the three
requirements can be satisfied simultaneously.
 Seth Gilbert and Nancy Lynch from MIT
proved the correctness of CAP theory in 2002.

 classified into three bottom-up levels:
1. File systems
2. Databases
3. Programming models.

1. File systems
 Google’s GFS, HDFS, Microsoft Cosmos,
Facebook Haystack,TaobaoTFS and FastDFS

1. File systems
2. Databases, NoSQL
1. Key-value Databases: Dynamo, Voldemort
2. Column-oriented Database: BigTable, Cassandra
3. Document Database: More complex with KV
MongoDB, SimpleDB, and CouchDB

1. File systems
2. Databases, NoSQL
3. Programming models
1.MapReduce: Automatic parallel processing. Sawzall
of Google, Pig Latin ofYahoo, Hive of Facebook,
and Scope of Microsoft.
2.Dryad: Dryad is a general-purpose distributed
execution engine for processing parallel
applications. And All-Pairs.

 Use proper statistical methods to analyze
massive data, to concentrate, extract, and
refine useful data hidden in a batch of chaotic
datasets, and to identify the inherent law of
the subject matter.

 Cluster Analysis: is a statistical method for
grouping objects, and specifically, classifying
objects according to some features.

 Factor Analysis: is basically targeted at
describing the relation among many
elements with only a few factors, i.e.,
grouping several closely related variables into
a factor, and the few factors are then used to
reveal the most information of the original
data.

 Correlation Analysis: is an analytical method
for determining the law of relations, such as
correlation, correlative dependence, and
mutual restriction, among observed
phenomena and accordingly conducting
forecast and control.

 Regression Analysis: is a mathematical tool
for revealing correlations between one
variable and several other variables.

 A/BTesting: also called bucket testing. It is a
technology for determining how to improve
target variables by comparing the tested
group. Big data will require a large number of
tests to be executed and analyzed.

 Statistical Analysis: Statistical analysis is
based on the statistical theory, a branch of
applied mathematics.

 Data Mining Algorithms: Data mining is a
process for extracting hidden, unknown, but
potentially useful information and knowledge
from massive, incomplete, noisy, fuzzy, and
random data.

 Bloom Filter: Bloom Filter consists of a series
of Hash functions.

 Hashing: it is a method that essentially
transforms data into shorter fixed-length
numerical values or index values.

 Index: index is always an effective method to
reduce the expense of disk reading and
writing, and improve insertion, deletion,
modification, and query speeds in both
traditional relational databases that manage
structured data, and other technologies that
manage semi structured and unstructured
data

 Triel: also called trie tree, a variant of Hash
Tree. It is mainly applied to rapid retrieval and
word frequency statistics.The main idea of
Triel is to utilize common prefixes of
character strings to reduce comparison on
character strings to the greatest extent, so as
to improve query efficiency.

 Parallel Computing: compared to traditional
serial computing, parallel computing refers to
simultaneously utilizing several computing
resources to complete a computation task.
 Presently, some classic parallel computing
models include MPI (Message Passing
Interface), MapReduce, and Dryad

1.Real-time vs. offline analysis:
Real-time is mainly used in E-commerce and
finance.
Offline analysis: is usually used for
applications without high requirements on
response time, e.g., machine learning,
statistical analysis, and recommendation
algorithms.

2.Analysis at different levels
Memory-level analysis: is for the case where the
total data volume is smaller than the maximum
memory of a cluster.
BI analysis: is for the case when the data scale
surpasses the memory level but may be imported
into the BI analysis environment.
Massive analysis: is for the case when the data
scale has completely surpassed the capacities of BI
products and traditional relational databases.

3. Analysis with different complexity

 R (30.7 %): R, an open source programming
language and software environment, is
designed for data mining/analysis and
visualization.
 Excel (29.8 %): Excel, a core component of
Microsoft Office, provides powerful data
processing and statistical analysis
capabilities.

 Rapid-I Rapidminer (26.7 %): Rapidminer is an
open source software used for data mining,
machine learning, and predictive analysis.
 KNMINE (21.8%): KNIME (Konstanz
Information Miner) is a user-friendly,
intelligent, and open-source rich data
integration, data processing, data analysis,
and data mining platform

 Weka/Pentaho (14.8 %):Weka, abbreviated
fromWaikato Environment for Knowledge
Analysis, is a free and open-source machine
learning and data mining software written in
Java.

 Application of big data in enterprises
 Application of IoT based big data
 Application of online social network-oriented
big data
 Application of online social network-oriented big
data
 Structure-based Applications
 EarlyWarning
 Real-time Monitoring
 Real-time Feedback

 Applications of healthcare and medical big
data

 Fundamental problems of big data:There is a
compelling need for a rigorous and holistic
definition of big data, a structural model of
big data, a formal description of big data.
 Standardization of big data: An evaluation
system of data quality and an evaluation
standard/benchmark of data computing
efficiency should be developed.

 Evolution of big data computing modes:This
includes memory mode, data flow mode,
PRAM mode, and MR mode, etc.

 Format conversion of big data
 Big data transfer
 Real-time performance of big data
 Processing of big data

 Big data management
 Searching, mining, and analysis of big data
 Integration and provenance of big data
 Big data application

 Big data privacy
 Protection of personal privacy during data acquisition
 Personal privacy data may also be leaked during storage,
transmission, and usage, even if acquired with the
permission of users
 Data quality: Data quality influences big data utilization.
Low quality data wastes transmission and storage
resources with poor usability
 Big data safety mechanism: Big data brings about
challenges to data encryption due to its large scale and
high diversity.

“Patience ensures
victory.”

Big data analytics, survey r.nabati

More Related Content

What's hot (20)

Similar to Big data analytics, survey r.nabati (20)

More from nabati (9)

Recently uploaded (20)

Big data analytics, survey r.nabati

Editor's Notes