Implementing Big Data, NoSQL, & Hadoop - Bigger Is (Usually) Better

Peter Aiken, Ph.D. & Micah Dalton
Implementing Big Data, NOSQL, & HADOOP
Demystifying Big Data: Bigger is (Usually) Better
Copyright 2017 by Data Blueprint Slide # 1
• DAMA International President 2009-2013
• DAMA International Achievement Award 2001 (with
Dr. E. F. "Ted" Codd
• DAMA International Community Award 2005
Peter Aiken, Ph.D.
• 33+ years in data management
• Repeated international recognition
• Founder, Data Blueprint (datablueprint.com)
• Associate Professor of IS (vcu.edu)
• DAMA International (dama.org)
• 10 books and dozens of articles
• Experienced w/ 500+ data
management practices
• Multi-year immersions: 
– US DoD (DISA/Army/Marines/DLA) 
– Nokia 
– Deutsche Bank 
– Wells Fargo 
– Walmart 
– … PETER AIKEN WITH JUANITA BILLINGS
FOREWORD BY JOHN BOTTEGA
MONETIZING
DATA MANAGEMENT
Unlocking the Value in Your Organization’s
Most Important Asset.
The Case for the
Chief Data Officer
Recasting the C-Suite to Leverage
Your MostValuable Asset
Peter Aiken and
Michael Gorman
2
Copyright 2017 by Data Blueprint Slide #

Micah Dalton
3Copyright 2017 by Data Blueprint Slide #
Micah is a senior business leader with twenty years of
management experience building and leading teams
to deliver results across various industries including;
financial services, public sector, non-profit and higher
education. Micah’s expertise in offering pragmatic
business solutions has made him valuable member of
client team. Micah's skills focus on using data to
drive root cause identification, analytics, strategy,
financial analysis and reporting, procurement strategy
and cost management, and operations analysis and
management. Micah helped lead the development of
Capital One’s Six Sigma program & completed his
Black Belt training. Micah also holds certifications in
Organizational Change Management (PROSCI) and
Data Management (CDMP-Associate from DAMA).
Micah earned his MBA from Duke’s Fuqua School of
Business focusing his interests in corporate finance
and business strategy. Prior to that Micah earned this
Bachelor’s degree in economics from Mary
Washington College. Additionally, Micah was a
member of the 2014 class of Leadership Metro
Richmond and has been an adjunct professor of
Marketing at the University of Mary Washington.
Implementing Big Data, NOSQL, & HADOOP 
• Why it is important to consider the messenger
– What is being "sold?"
– We are using the wrong vocabulary to discuss this topic
• Technically what are Big Data Technologies good at?
– Computers→ commodity-based computing infrastructure
– Flash memory is currently obeying Moore's Law
– RAM→increased processing
– Parallel-friendly approaches (lots of repeatable actions)
• Successful Big Data Approaches ...
– Innovation
– Reengineering (precise definition)
– Throw away Prototyping
• How does that help operationally?
– Solid support community
– Examples
– Innovation
– Examples

Welcome to the Post-Big Data Era!
Data Velocity
Data Volume
Data Variety
Big Data: Expanding on 3
Fronts at an Increasing Rate
Big Data(has something to do with Vs - doesn't it?)
• Volume
– Amount of data
• Velocity
– Speed of data in and out
• Variety
– Range of data types and sources
• 2001 Doug Laney
• Variability
– Many options or variable interpretations confound analysis
• 2011 ISRC
• Vitality
–A dynamically changing Big Data environment in which analysis and predictive models
must continually be updated as changes occur to seize opportunities as they arrive
• 2011 CIA
• Virtual
– Scoping the discussion to only include online assets
• 2012 Courtney Lambert
• Value/Veracity
• Stuart Madnick (John Norris Maguire Professor of Information Technology, MIT Sloan School of
Management & Professor of Engineering Systems, MIT School of Engineering)

The 13 V’s of Big Data
• Vast Volume of Vigorously, Verified, Vexingly, Variable,
Verbose yet Valuable, Vital, Visualized, high Velocity and
Veracity data that encourages the Vanity of the big data
experts
– Original from John Marshey – Sillicon Graphics 1998 
(with contributed extensions)
• We have no objective
definition of big data!
– Any measurements,
claims of success,
quantifications, etc.
must be viewed
skeptically and with
suspicion!
– Innovation
– Examples
– Innovation
– Examples

I shall not today
attempt further to
define the kinds of
material but I know
it when I see it ...
(Justice Potter Stewart)
Big Data

Big Data
[ Techniques /
Technologies ]
Big Data

Big Data Techniques
• New techniques available to impact the productivity (order of
magnitude) of any analytical insight cycle that compliment,
enhance, or replace conventional (existing) analysis methods
• Big data techniques are currently characterized by:
– Continuous, instantaneously  
available data sources
– Non-von Neumann  
Processing (defined later in the presentation)
– Capabilities approaching  
or past human comprehension
– Architecturally enhanceable  
identity/security capabilities
– Other tradeoff-focused data processing
• So a good question becomes "where in our existing architecture
can we most effectively apply Big Data Techniques?"
The Big Data Landscape
Copyright Dave Feinleib, bigdatalandscape.com

The Big Data Landscape 2.0
The Big Data Landscape 3.0
Copyright Dave Feinleib, bigdatalandscape.com

Internet of Things Landscape 2016
http://blogs.cisco.com/sp/from-internet-of-things-to-web-of-things/

Howmuchdatais 
generatedeveryminute!
114#
Rela%onal(zone(
Non+rela%onal(zone(
Lotus#Notes#
Objec/vity#
MarkLogic#
InterSystems#
Caché#
McObject#
Starcounter#
ArangoDB#
Founda/onDB#
Neo4J#
InfiniteGraph#
CouchDB#
Oracle#NoSQL#
Redis#
Handlersocket#
##RavenDB#
RethinkDB#
Google#App##
Engine#Datastore#
LevelDB#
Accumulo#
Cassandra#
HBase#
Riak#
Couchbase#
Splice#Machine#
Ac/an#Ingres#
SAP#Sybase#ASE#
EnterpriseDB#
SQL##
Server#
MySQL#
Informix#MariaDB#
SAP##
HANA#
#
IBM#
DB2#
Database.com#
ClearDB#
Google#Cloud#SQL#
Rackspace#
Cloud#Databases#
AWS#RDS#
Azure#SQL#
Database#
FathomDB#
HP#Cloud#RDB#
#for#MySQL#
StormDB#
Teradata##
Aster#
HPCC#
Cloudera#
Hortonworks#MapR# IBM##
BigInsights#
ZeWaset#
NGDATA#
Infochimps#
Metascale#
Mortar#
Data#
Rackspace#
Qubole#
Voldemort#
Aerospike#
Teradata#
IBM#PureData#
for#Analy/cs#
Pivotal#Greenplum#
HP#Ver/ca#
SAP#Sybase#IQ#
IBM#InfoSphere#
Ac/an#Vector#
XtremeData#
Kx#Systems#
Exasol#
Ac/an#Matrix#
ParStream#
TokuDB#
ScaleDB#
ScaleArc#
Con/nuent#
TransLa[ce#
NuoDB#
Drizzle#
JustOneDB#
Pivotal#GemFire#XD#
Galera#
ScaleBase#
Zimory#Scale#
Clustrix#
Tesora#
MemSQL# GenieDB#
Datomic#YarcData#
FlockDB#
Allegrograph#
HypergraphDB#
AffinityDB#
Giraph#
Trinity# MemCachier#
Redis#Labs#
Memcached#Cloud#
FairCom#
BitYota#
IronCache#
Grid/cache(zone(
Memcached#
Ehcache#
ScaleOut#
So^ware#
IBM##
eXtreme##
Scale#
Oracle##
Coherence#
GigaSpaces#XAP#GridGain#
Pivotal#
GemFire#
CloudTran#
InfiniSpan#
Hazelcast#
Oracle#
Exaly/cs#
Oracle#
Database#
MySQL#Cluster#
Oracle##
Endeca#Server# A[vio#
Elas/csearch#
LucidWorks#
Big#Data#
Lucene/Solr#
IBM#InfoSphere##
Data#Explorer#
Towards(
E*discovery(
Towards(
enterprise(search(
Documentum#
xDB#
Tamino#
XML#Server#
Ipedo#XML#
Database#
ObjectStore#
LucidDB#
MonetDB#
Metamarkets#Druid#
Databricks/Spark#
AWS#
Elas/Cache#
#
Firebird#
SciDB#
SQLite#
Oracle#TimesTen#
solidDB#
Adabas#
IBM#IMS#
UniData#
UniVerse#
WakandaDB#
Al/scale#
Oracle#Big##
Data#Appliance#
RainStor#
OrientDB#
Sparksee#
Metamarkets#
Treasure#
Data#
PostgreSQL#
Percona#
vFabric#Postgres#
©#2014#by#451#Research#LLC.##
All#rights#reserved##
HyperDex#
TIBCO#
Ac/veSpaces#
Titan#
SAP#Sybase#SQL#Anywhere#
JethroData#
CitusDB#
Pivotal#
HD/HAWQ#
BigMemory#
Ac/an#
Versant#
DataStax#
Enterprise#
DeepDB#
Infobright#
FatDB#
Google#Cloud#
Datastore#
Heroku#
Postgres#
GrapheneDB#
Cassandra.io#
Hypertable#
BerkeleyDB#
Sqrrl#
Enterprise#
Microso^#
HDInsight#
HP#
Autonomy#
Oracle#
Exadata#
IBM##
PureData#
IBM#
Big#SQL#
Impala#
Apache#
Drill#
Presto#
Microso^#
SQL#Server#
PDW#
Apache#
Tajo#
Apache#
Hive#
SPARQLBASE#
MammothDB#
Al/base#HDB#
LogicBlox#
SRCH2#
TIBCO#
LogLogic#
Splunk#
Towards(
SIEM(
Loggly# Sumo#
Logic#Logentries#
InfiniSQL#
JumboDB#
Ac/an#PSQL#
Progress#OpenEdge#
Kogni/o#
Al/base#XDB#
Savvis#
So^layer#
Verizon#
xPlenty#
Stardog#
MariaDB#Enterprise#
Apache#Storm#
Apache#S4#
IBM#
InfoSphere#
Streams#
TIBCO#
StreamBase#
DataTorrent#
AWS#
Kinesis#
Feedzai#
Guavus#
Lokad#
SQLStream#
So^ware#AG#
Key:((
General#purpose#
Specialist#analy/c#
BigTables#
Graph#
Document#
Key#value#stores#
easeaeService#
Key#value#direct##
access#
Hadoop#
MySQL#ecosystem#
Advanced##
clustering/sharding#
New#SQL#databases#
Data#caching#
Data#grid#
Search#
Appliances#
Inememory#
Stream#processing#
OpenStack#Trove#
1010data#
Google##
BigQuery#
AWS#
Redshi^#
TempoIQ#
InfluxDB#
WebScaleSQL#
MySQL#
Fabric#Spider#
2#
E#
D
A
B
C
TeSystems#
E#
D
A
B
C
2# 4#3# 5#
SQream#
SpaceCurve#
PostgreseXL#
Google#Cloud##
Dataflow#
Trafodion# Hadapt#
Azure#
Search#
Red#Hat#JBoss#
Data#Grid#
6#5#4#
MongoDB#
Cloudant#
Iris#Couch#
MongoLab#
Compose#
ObjectRocket#
CloudBird#
Azure#DocumentDB#
1# 3#
1# 6#
Data
Platforms
Map
October 2014
https://
451research.com/
dashboard/dpa
CockroachDB#
AWS#DynamoDB#
Redisetoego#
AWS#SimpleDB#
Redis#Labs#
Redis#Cloud#
RedisGreen#
AWS#Elas/Cache#
with#Redis#
MagnetoDB#
ObjectRocket#
Redis#
##TokuMX#
VoltDB#
CortexDB#
CodeFutures#
Oracle#Big##
Data#Cloud#
AWS#
EMR#
Google##
Compute#
Engine#
Stra/o#

INDEX#
D6 #1010data#
D2 #Accumulo#
B3 #Ac/an#Ingres#
C6 #Ac/an#Matrix#
B5 #Ac/an#PSQL#
C6 #Ac/an#Vector#
E1 #Ac/an#Versant#
D1 #Adabas#
C2 #Aerospike#
E1 #AffinityDB#
E1 #Allegrograph#
D3 #Al/base#HDB#
D3 #Al/base#XDB#
A3 #Al/scale#
B4 #Apache#Drill#
B4 #Apache#Hive#
A2 #Apache#S4#
A2 #Apache#Storm#
B3 #Apache#Tajo#
B2 #ArangoDB#
A1 #A[vio#
E2 #AWS#DynamoDB#
E4 #AWS#Elas/Cache#
E2 #AWS#Elas/Cache#with#Redis#
A4 #AWS#EMR#
A2 #AWS#Kinesis#
D5 #AWS#RDS#
D6 #AWS#Redshi^#
E2 #AWS#SimpleDB#
E2 #Azure#DocumentDB#
B2 #Azure#Search#
D5 #Azure#SQL#Database#
D2 #BerkeleyDB#
E4 #BigCache#
E4 #BigMemory#
D6 #BitYota#
C2 #Cassandra#
D2 #Cassandra.io#
B5 #CitusDB#
D5 #ClearDB#
E2 #Cloudant#
D2 #CloudBird#
A5 #Cloudera#
E5 #CloudTran#
C4 #Clusrix#
C3 #CockroachDB#
C4 #CodeFutures#
D2 #Compose#
D4 #Con/nuent#
B2 #CortexDB#
C2 #Couchbase#
D2 #CouchDB#
D5 #Database.com#
A5 #Databricks/Spark#
C2 #DataStax#Enterprise#
#
A2 #DataTorrent#
C3 #Datomic#
D4 #DeepDB#
C1 #Documentum#xDB#
C4 #Drizzle#
E5 #Ehcache#
A1 #Elas/csearch#
B3 #EnterpriseDB#
C4 #CodeFutures#
C4 #CodeFutures#
E2 #Compose#
D4 #Con/nuent#
C2 #Couchbase#
D2 #CouchDB#
D5 #Database.com#
A5 #Databricks/Spark#
C2 #DataStax#Enterprise#
A2 #DataTorrent#
C3 #Datomic#
D4 #DeepDB#
E2 #DocumentDB#
C1 #Documentum#xDB#
C5 #Drizzle#
E5 #Ehcache#
A1 #Elas/csearch#
B3 #EnterpriseDB#
C5 #Exasol#
C3 #FairCom#
C2 #FatDB#
D5 #FathomDB#
A2 #FeedZai#
B3 #Firebird#
D1 #FlockDB#
C2 #Founda/onDB#
D4 #Galera#
C4 #GenieDB#
E4 #GigaSpaces#XAP#
E1 #Giraph#
D5 #Google#BigQuery#
D2 #Google#App#Engine#Datastore#
A2 #Google#Cloud#Dataflow#
D2 #Google#Cloud#Datastore#
C5 #Google#Cloud#SQL#
A4 #Google#Compute#Engine#
D1 #GrapheneDB#
E3 #GridGain#
A2 #Guavus#
B5 #Hadapt#
C2 #Handlersocket#
E5 #Hazelcast#
C2 #HBase#
C5 #Heroku#Postgres#
A5 #Hortonworks#
A1 #HP#Autonomy#
D5 #HP#Cloud#RDB#for#MySQL#
C6 #HP#Ver/ca#
#
B6 #HPCC#
D2 #HyperDex#
E1 #HypergraphDB#
C2 #Hypertable#
B4 #IBM#Big#SQL#
A5 #IBM#BigInsights#
B4 #IBM#DB2#
E6 #IBM#eXtreme#Scale#
D1 #IBM#IMS#
C6 #IBM#InfoSphere#
B2 #IBM#InfoSphere#Data#Explorer#
A2 #IBM#InfoSphere#Streams#
B4 #IBM#PureData#
B6 #IBM#PureData#for#Analy/cs#
B5 #Impala#
E6 #InfiniSpan#
C3 #InfiniSQL#
E1 #InfiniteGraph#
D6 #InfluxDB#
C4 #Infobright#
A3 #Infochimps#
B5 #Informix#
E1 #Intersystems#Caché#
C1 #Ipedo#XML#Database#
E2 #Iris#Couch#
E4 #IronCache#
B5 #JethroData#
D2 #JumboDB#
C3 #JustOneDB#
C6 #Kogni/o#
C6 #Kx#Systems#
D2 #LevelDB#
B1 #Logentries#
B1 #Loggly#
D6 #LogicBlox#
A2 #Lokad#
E2 #Lotus#Notes#
A1 #Lucene/Solr#
C6 #LucidDB#
B2 #LucidWorks#Big#Data#
E2 #MagnetoDB#
B4 #MammothDB#
A4 #MapR#
B3 #MariaDB#
B3 #MariaDB#Enterprise#
B2 #MarkLogic#
D1 #McObject#
E5 #Memcached#
E3 #MemCachier#
C3 #MemSQL#
A3 #Metamarkets#
C6 #Metamarkets#Druid#
A5 #Metascale#
A5 #Microso^#HD#Insight#
B5 #Microso^#SQL#Server##
B5 #Microso^#SQL#Server#PDW#
#
#
#
#
D6 #MonetDB#
D2 #MongoDB#
E2 #MongoLab#
A3 #Mortar#Data#
B4 #MySQL#
C4 #MySQL#Cluster#
C4 #MySQL#Fabric#
C1 #Neo4J#
B2 #NGDATA#
C3 #NuoDB#
E1 #Objec/vity#
E2 #ObjectRocket#
D2 #ObjectRocket#Redis#
D1 #ObjectStore#
C5 #OpenStack#Trove#
A5 #Oracle#Big#Data#Appliance#
A5 #Oracle#Big#Data#Cloud#
E5 #Oracle#Coherence#
B4 #Oracle#Database#
A1 #Oracle#Endeca#Server#
B4 #Oracle#Exadata#
B6 #Oracle#Exaly/cs#
D2 #Oracle#NoSQL#
C5 #Oracle#TimesTen#
C1 #OrientDB#
C6 #ParStream#
B3 #Percona#
E4 #Pivotal#GemFire#
D6 #Pivotal#Greenplum#
B5 #Pivotal#HD/HAWQ#
D3 #Pivotal#SQLFire#
B3 #PostgreseXL#
B3 #PostgreSQL#
B4 #Presto#
C5 #Progress#OpenEdge#
A3 #Qubole#
A3 #Rackspace#
C5 #Rackspace#Cloud#Databases#
B6 #RainStor#
D2 #RavenDB#
E6 #Red#Hat#JBoss#Data#Grid#
C2 #Redis#
E3 #Redis#Labs#Memcached#Cloud#
E2 #Redis#Labs#Redis#Cloud#
E2 #Redisetoego#
E2 #RedisGreen#
D2 #RethinkDB#
C2 #Riak#
B5 #SAP#HANA#
B3 #SAP#Sybase#ASE#
C6 #SAP#Sybase#IQ#
B3 #SAP#Sybase#SQL#Anywhere#
A3 #Savvis#
C4 #ScaleArc#
C4 #ScaleBase#
C4 #ScaleDB#
#
E3 #ScaleOut#So^ware#
B6 #SciDB#
A3 #So^layer#
A2 #So^ware#AG#
C5 #solidDB#
D6 #SpaceCurve#
C1 #Sparksee#
E1 #SPARQLBASE#
C4 #Spider#
B3 #Splice#Machine#
B2 #Splunk#
B3 #SQLite#
A2 #SQLStream#
B6 #SQream#
B2 #Sqrrl#Enterprise#
A1 #SRCH2#
B2 #Starcounter#
D1 #Stardog#
C5 #StormDB#
A6 #Stra/o#
B1 #Sumo#Logic#
A3 #TeSystems#
C1 #Tamino#XML#Server#
D6 #TempoIQ#
B6 #Teradata#
B6 #Teradata#Aster#
C4 #Tesora#
E4 #TIBCO#Ac/veSpaces#
B1 #TIBCO#LogLogic#
A2 #TIBCO#StreamBase#
D1 #Titan#
C4 #TokuDB#
D2 #TokuMX#
B3 #Trafodion#
D3 #TransLa[ce#
A4 #Treasure#Data#
E1 #Trinity#
C1 #UniData#
C1 #UniVerse#
A3 #Verizon#
B3 #vFabric#Postgres#
D2 #Voldemort#
C3 #VoltDB#
D1 #WakandaDB#
D5 #WebScaleSQL#
A3 #xPlenty#
B6 #XtremeData#
C1 #YarcData#
A4 #ZeWaset#
D4 #Zimory#Scale#
#
#
#
hWps://451research.com/dashboard/dpa#
Big Data = Big Spending
• Enterprises are spending wildly on Big Data but don’t
know if it’s worth it yet (Business Insider, 2012)
• Big Data Technology Spending Trend:
• 83% increase over the next 3 years (worldwide):
– 2012: $28 billion
– 2013: $34 billion
– 2016: $232 billion
• Caution:
– Don’t fall victim to SOS (Shiny Object  
Syndrome)
– A lot of money is being invested but  
is it generating the expected return?
– Gartner Hype Cycle suggests results  
are going to be disappointing
http://www.businessinsider.com/enterprise-big-data-spending-2012-11#ixzz2cdT8shhe
http://www.inc.com/kathleen-kim/big-data-spending-to-increase-for-it-industry.html
http://www.gartner.com/DisplayDocument?id=2195915&ref=clientFriendlyUrl

Big Data Technologies by themselves, are a One Legged Stool
Governance is the major means
of preventing over reliance on
one legged stools!
– Innovation
– Examples
– Innovation
– Examples

Costpercomputingcycledeclining
– Innovation
– Examples
– Innovation
– Examples

10X+++ rapid access
"There’s now a blurring between the storage world and the memory world"
• Faster processors outstripped
not only the hard disk, but main
memory
– Hard disk too slow
– Memory too small
• Flash drives remove both
bottlenecks
– Combined Apple and Yahoo have
spend more than $500 million to
date
• Make it look like traditional
storage or more system
memory
– Minimum 10x improvements
– Dragonstone server is 3.2 tb flash
memory (Facebook)
• Bottom line - new capabilities!

Non-von Neumann Processing/Efficiencies
• von Neumann
bottleneck  
(computer science)
– "An inefficiency inherent in
the design of any von
Neumann machine that
arises from the fact that
most computer time is
spent in moving
information between
storage and the central
processing unit rather than
operating on it"  
[http://encyclopedia2.thefreedictionary.com/von+Neumann+bottleneck]
• Michael Stonebraker
– Ingres (Berkeley/MIT)
– Modern database
processing is
approximately 4%
efficient
• Many big data
architectures are
attempts to address
this, but:
– Zero sum game
– Trade characteristics
against each other
• Reliability
• Predictability
– Google/MapReduce/
Bigtable
– Amazon/Dynamo
– Netflix/Chaos Monkey
– Hadoop
– McDipper
• Big data techniques
exploit non-von
Neumann processing
30
What is NoSQL?
• Commonly interpreted as both "No SQL" and "Not Only SQL
• Broad class of database management technologies that
provide a mechanism for storage and retrieval of data that
doesn’t follow traditional relational database methodology.
• Motivations
– Simplicity of design
– Horizontal scaling
– Finer control over availability of the data.
• The data structures used by NoSQL databases differ from
those used in relational databases, making some operations
faster in NoSQL and others  
faster in relational  
databases

What is Hadoop?
• A data storage and processing  
system, that runs on clusters of commodity servers.
• Able to store any kind of data in its native format.
• Perform a wide variety of analyses and transformations.
• Store terabytes, and even petabytes, of data
inexpensively.
• Handles hardware and system failures automatically,
without losing data or interrupting data analyses.
• Critical components of Hadoop:
– HDFS- The Hadoop Distributed File System is the storage system
for a Hadoop cluster, responsible for distribution of data across the
servers.
– Mapreduce- The inner workings of Hadoop that allows for distributed
and parallel analytical job execution.
One of Data Blueprint's Big Data Clusters

Why NoSQL? Why Hadoop?
• Large number of users (read: the internet)
• Rapid app development and deployment
• Large number of mission critical writes (sensors/etc)
• Small, continuous reads and writes, especially where
“Consistency” is less important (social networks)
• Hadoop solves the hard scaling problems caused by large
amounts of complex data.
• As the amount of data in a cluster grows,  
new servers can be added to a Hadoop  
cluster incrementally and inexpensively  
to store and analyze it.
Hadoop Use Cases in the Real World
• Risk Modeling
• Customer Churn Analysis
• Recommendation Engine
• Ad Targeting
• Point of Sale Transaction Analysis
• Social Sentiment on Social Media
• Analyzing network data to predict failure
• Threat analysis
• Trade Surveillance

http://blogs.informatica.com/perspectives/uk/2011/08/09/hadoop-enriches-data-science-part-2-of-hadoop-series/
Potential Tradeoffs:
CAP theorem: consistency, availability and partition-tolerance
 
 
 
Partition  
(Fault) 
Tolerance
 
Availability
 
Consistency
RDBMS
NOSQL
Atomicity
Consistency
Isolation
Durability
Basic
Availability
Soft-state
Eventual consistency

– Innovation
– Examples
– Innovation
– Examples
Pacman
• Decomposition
• Reassembly
– not optional!

– Innovation
– Examples
– Innovation
– Examples
Sandwich use case
• Landing Zone (less
expensive)
– Especially useful in cases were
data is highly disposable
• Existing technologies are the
– Contents sandwiched and  
complemented landing zone and
archival capabilities
• Archiving/Offloading (less
need for structure)
– "Cold" transactional and analytic
data 
Adapted from Nancy Kopp:  
http://ibmdatamag.com/2013/08/relishing-the-big-data-burger/
Landing_Zone
Archiving_Offloading
Existing  
Data Architectural  
Processing

See Like a Snake

Pit Organ
They can switch back and forth
between those two systems, or
use both simultaneously, giving
them a leg up, so to speak,
when it comes to targeting a
warm object.
Pit Organ
– Innovation
– Examples
– Innovation
– Examples

<-Feedback
Discernm
ent
 
Exploitable 
Insight
• Patterns/objects,
hypotheses emerge
– What can be observed?
• Operationalizing
– The dots can be
repeatedly connected
Analytics Insight Cycle
!
Exis&ng!
Knowledge
/base
• Things are happening
– Sensemaking
techniques address
"what" is happening?
• Patterns/objects,
hypotheses emerge
– What can be observed?
• Operationalizing
– The dots can be
repeatedly connected
– "Big Data" contributions
are shown in orange
• Margaret Boden's
computational
creativity
– Exploratory
– Combinational
– Transformational
Volume
Velocity
Variety
 
 
Potential/
actual
insights
Pattern/Object 
Emergence  
Analytical
bottleneck 
 
 
 
 
 
 
 C
om
bined/
inform
ed
insights
"Sensemaking"  
Techniques
Humans Generally Better Machines Generally Better
• Sense low level stimuli
• Detect stimuli in noisy background
• Recognize constant patterns in varying situations
• Sense unusual and unexpected events
• Remember principles and strategies
• Retrieve pertinent details without a priori connection
• Draw upon experience and adapt decision to situation
• Select alternatives if original approach fails
• Reason inductively; generalize from observations
• Act in unanticipated emergencies and novel situations
• Apply principles to solve varied problems
• Make subjective evaluations
• Develop new solutions
• Concentrate on important tasks when overload occurs
• Adapt physical response to changes in situation
• Sense stimuli outside human's range
• Count or measure physical quantities
• Store quantities of coded information accurately
• Monitor prespecified events, especially infrequent
• Make rapid and consisted responses to input signals
• Recall quantities of detailed information accurately
• Retrieve pertinent detailed without a priori connection
• Process quantitative data in prespecified ways
• Perform repetitive preprogrammed actions reliably
• Exert great, highly controlled physical force
• Perform several activities simultaneously
• Maintain operations under heavy operation load
• Maintain performance over extended periods of time
J. C. R. Licklider's Man-Computer Symbiosis
Best approaches combines manual and automated methods!

– Innovation
– Examples
– Innovation
– Examples
Gartner Recommendations
Impacts Top
RecommendationsSome of the new analytics that are made
possible by big data have no precedence,
so innovative thinking will be required to
achieve value
Treat big data projects as innovation
projects that will require change
management efforts. The business will
take time to trust new data sources and
new analytics
Creative thinking can unearth valuable
information sources already inside the
enterprise that are underused
Work with the business to conduct an
inventory of internal data sources outside
of IT's direct control, and consider
augmenting existing data that is IT
'controlled.' With an innovation mindset,
explore the potential insight that can be
gained from each of these sources
Big data technologies often create the
ability to analyze faster, but getting value
from faster analytics requires business
changes
Ensure that big data projects that improve
analytical speed always include a process
redesign effort that aims at getting
maximum benefit from that speed
Gartner 2012

Innovation
• Innovation is the development of new customers
value through solutions that meet new needs,
inarticulate needs, or old customer and market
needs in new ways. This is accomplished through
different or more effective products, processes,
services, technologies, or ideas that are readily
available to markets, governments, and society.
• Innovation differs from invention in that innovation
refers to the use of a better and, as a result, novel
idea or method, whereas invention refers more
directly to the creation of the idea or method itself.
• Innovation differs from improvement in that
innovation refers to the notion of doing something
different (Lat. innovare: "to change") rather than
doing the same thing better.
Data must be incorporated into the innovation-navigation process

Two Uses for Data in Support of Innovation
1. Using data to keep
the innovation
process on track
2. Using data to
innovate
Q3 Q4 Q1 Q2
Innovation Series Kick Off Performance Reviews
Intro to Innovative
Thinking & Strategies
Incremental Innovation Disruptive Innovation
INSPIRATION
TOOLS
INFRASTRUCTURE
ACCOUNTABILITY
Think Different Series, Inspired Reading, Innovation Experiences
Desk Drops, IS Innovation, Academy Classes
Town Hall Meetings, Break Out Rooms, Innovation Board
Performance Reviews
Big Data Display
- Game Plan 2013 Session
- Big Data (Speaker TBD)
Think Different Session
- Innovation
- Negotiation
- Providing Feedback that Inspires
*
ALTRIA IS INNOVATION TRANSFORMATION 2012 - 2013
IS INNOVATION ECOSYSTEM
Tech Display
- Town Hall Session
“Creating an Innovation Ecosystem”
Evolution of Publishing Display
- Think Different Session:
Jason Ashlock
- Effective Communication
- Strategic Agility
- Cultivating an Innovation Culture
*
iGloo Room Displays
- Speaker Session
- IS Innovation Academy
* managers only
KEY
Experiential Component
Ongoing Communications
Q3
Sustaining
Transformational Commercial Disruptive“Er”
© THE FRONTIER PROJECT, LLC
Data
Data
Data
Data
52

– Innovation
– Examples
– Innovation
– Examples
Reengineering(Objective Definition)
• How can state that you
have improved any
system?
• If you don't understand
the existing (legacy)
systems strengths and
weaknesses
• You can't use that
these to inform the new
system
• To reengineer
– You must first reverse
engineering and then
– Use that information to
architect the new system
Legacy System Analysis 
(break down & compare)
$$$Value
New System Requirements
New System

– Innovation
– Examples
– Innovation
– Examples
Copyright 2013 by Data Blueprint
Potential Tradeoffs:
CAP theorem: consistency, availability and partition-tolerance
56
 
 
 
Partition  
(Fault) 
Tolerance
 
Availability
 
Consistency
RDBMS
NOSQL
Small datasets can be both consistent & available
Atomicity
Consistency
Isolation
Durability
Basic
Availability
Soft-state
Eventual consistency

'Throw-away' prototyping
• With 'throw-away' prototyping a small
part of the system is developed and
then given to the end user to try out
and evaluate. The user provides
feedback which can quickly be
incorporated into the development of
the main system. The prototype is
then discarded or thrown away.
– Innovation
– Examples
– Innovation
– Examples

David Brooks, New York Times
59
• Data analysis struggles with the social
– Your brain is excellent at social cognition - people can
• Mirror each other’s emotional states
• Detect uncooperative behavior
• Assign value to things through emotion
– Data analysis measures the quantity of social
interactions but not the quality
• Map interactions with co-workers you see during work days
• Can't capture devotion to childhood friends seen annually
– When making (personal) decisions about social
relationships, it’s foolish to swap the amazing machine
in your skull for the crude machine on your desk
• Data struggles with context
– Decisions are embedded in sequences and contexts
– Brains think in stories - weaving together multiple
causes and multiple contexts
– Data analysis is pretty bad at
• Narratives / Emergent thinking / Explaining
• Data creates bigger haystacks
– More data leads to more statistically significant
correlations
– Most are spurious and deceive us
– Falsity grows exponentially greater amounts of data
we collect
• Big data has trouble with big problems
– For example: the economic stimulus debate
– No one has been persuaded by data to switch sides
• Data favors memes over masterpieces
– Detect when large numbers of people take an instant
liking to some cultural product
– Products are hated initially because they are unfamiliar
• Data obscures values
– Data is never raw; it’s always structured according to
somebody’s predispositions and values
Some Big Data Limitations
Maslow's Hierarchiy of Needs

You can accomplish
Advanced Data Practices
without becoming proficient
in the Foundational Data
Practices however  
this will:
• Take longer
• Cost more
• Deliver less
• Present  
greater 
risk (with thanks to Tom DeMarco)
Data Management Practices Hierarchy
Advanced  
Data  
Practices
• MDM
• Mining
• Big Data
• Analytics
• Warehousing
• SOA
Foundational Data Practices
Data Platform/Architecture
Data Governance Data Quality
Data Operations
Data Management Strategy
Technologies
Capabilities
– Innovation
– Examples
– Innovation
– Examples

Social Sentiment Analysis
• One of the burgeoning areas
for use of Big Data / Hadoop
platforms.
• Allows for the landing of
multiple sources of
unstructured data. (Twitter,
Facebook, Linked In, etc.)
• Data than can be analyzed
with algorithms looking for
keywords that determine
positive/negative feedback
63
64
Operational Use
• Utilize real time pricing data from multiple sources to dynamically
update the pricing for books in the Amazon Marketplace.
• Ingested data from multiple sources looking for real time changes
in price.
• Would apply predictive model to determine best price point and set
price of their books on the marketplace.
• Increased conversion rate, but created a race to the bottom
situation if not monitored

65
Healthcare Example: Patient Data
• Clinical data:
– Diagnosis/prognosis/treatment
– Genetic data
• Patient demographic data
• Insurance data:
– Insurance provider
– Claims data
• Prescriptions & pharmacy information
• Physical fitness data
– Activity tracking through  
smartphone apps & social media
• Health history
• Medical research data
66
http://www.forbes.com/sites/xerox/2013/09/27/big-data-boosts-customer-loyalty-no-really/
Retail Example: Loyalty Programs & Big Data
• Companies need to understand current wants and needs AND
predict future tendencies
• Customer -> Repeat Customer -> Brand Advocate
• Customer loyalty programs & retention strategies
– Track what is being purchased and how often
– Coupons based on purchasing history
– Targeted communications, campaigns & special offers
– Social media for additional interactions
– Personalize consumer interactions
• Customer purchase history influences
product placements
– Retailers rapidly respond to consumer demands
– Product placements, planogram optimization, etc.

67
References
• The Human Face of Big Data, Rick Smolan & Jennifer Erwitt, First Edition edition (November
20, 2012)
• McKinsey: Big Data: The next frontier for innovation, competition and productivity
(http://www.mckinsey.com/insights/business_technology/
big_data_the_next_frontier_for_innovation?p=1)
• The Washington Post: Five Myths about Big Data (http://articles.washingtonpost.com/
2013-08-16/opinions/41416362_1_big-data-data-crunching-marketing-analytics)
• Gartner: Gartner’s 2013 Hype Cycle for Emerging Technologies Maps Out Evolving
Relationship Between Humans and Machines (http://www.gartner.com/newsroom/id/
2575515)
• The New York Times | Opinion Pages: What Data Can’t Do (http://www.nytimes.com/
2013/02/19/opinion/brooks-what-data-cant-do.html?_r=1&)
• CIO.com: Five Steps for How to Better Manage Your Data (http://www.cio.com.au/article/
429681/five_steps_how_better_manage_your_data/)
• Business Insider: Enterprises Aren’t Spending Wildly on ‘Big Data’ But Don’t Know If It’s
Worth It Yet (http://www.businessinsider.com/enterprise-big-data-
spending-2012-11#ixzz2cdT8shhe)
• Inc.com: Big Data, Big Money: IT Industry to Increase Spending (http://www.inc.com/
kathleen-kim/big-data-spending-to-increase-for-it-industry.html)
• Forbes: Big Data Boosts Customer Loyalty. No, Really. (http://www.forbes.com/sites/xerox/
2013/09/27/big-data-boosts-customer-loyalty-no-really/)
It’s your turn!
Use the chat feature or Twitter (#dataed) to submit
your questions to everyone now
68
Questions?

10124 W. Broad Street, Suite C
Glen Allen, Virginia 23060
804.521.4056

Implementing Big Data, NoSQL, & Hadoop - Bigger Is (Usually) Better

More Related Content

What's hot (20)

Similar to Implementing Big Data, NoSQL, & Hadoop - Bigger Is (Usually) Better (20)

More from DATAVERSITY (20)

Recently uploaded (20)

Implementing Big Data, NoSQL, & Hadoop - Bigger Is (Usually) Better