Cassandra + Hadoop @ApacheCon

 BigTable + Dynamo
 Semi-structured data model
 Decentralized – no special roles
 Ridiculously fast writes, fast reads
 Tunably consistent
 Cross-DC capable

 You design your data model based off of your
query model
 Real-time ad-hoc queries aren’t viable
 Secondary indexes help (0.7)
 What about analytics?

 Hadoop has analytics
 MapReduce
 Pig/Hive and other tools built above MapReduce
 Configurable data sources/destinations
 Many already familiar with it
 Active community

 Always able to output to Cassandra directly
 0.6
 ColumnFamilyInputFormat
 Pig support – Cassandra LoadFunc
 0.7
 ColumnFamilyOutputFormat
 Hadoop Streaming Output
 Streamlined configuration

 Recipe
 Overlay Hadoop on top of Cassandra
 Separate server for name node and job tracker
 Co-locate task trackers with Cassandra nodes
 Add data nodes to taste
 Voilà
 Data locality
 Analytics engine scales with data
 Example

 Cassandra specific InputFormat
 Configuration – ConfigHelper, Hadoop variables
 InputSplits over the data – tunable
 Example usage in contrib/word_count

 OutputFormat
 Configuration – ConfigHelper, Hadoop variables
 Batches output – tunable
 Don’t have to use Cassandra api
 Some optimizations (e.g. ConsistencyLevel.ONE)
 Example usage in contrib/word_count

 60,000+ Documented UFO Sightings
 Data set from http://infochimps.com
sighted_at reported_at location shape duration description
19951009 19951009 Iowa City, IA
Man repts.Witnessing “flash,
followed by a classic UFO, w/ a
tailfin at back.” …
19940801 19950220 Renton, WA
Man repts. seeing 2x large
ships hovering in night sky
while using Russian-made
night binoculars.
19970111 19970111 St. Cloud, MN pyramid 2 min.
Summary : Right when me and
my friend left my house we
saw a bright green glowing
object that looked like a 4
sided pyramid then after about
2 min it took off straight into
the sky leaving a yellow trail
behind it…

 What about languages outside of Java?
 Build on what Hadoop uses - Streaming
 Output streaming in 0.7.0
 Example in contrib/hadoop_streaming_output
 Input streaming in progress, likely 0.7.1

 Developed atYahoo!
 PigLatin/Grunt shell
 Powerful scripting language for analytics
 Example usage in contrib/pig
 Configuration – Hadoop/Env variables

 Raptr.com
 Home grown solution -> Cassandra + Hadoop
 Query time: hours -> minutes
 Pig obviated their need for multi-lingual MR
 Speed and ease are enabling
 Imagini/Visual DNA
 US Government (Digital Reasoning)
 See http://github.com/digitalreasoning/PyStratus

 Hive support in progress (HIVE-1434)
 Hadoop Input Streaming (likely 0.7.1)
 Performance improvements

 Hadoop analytics for Cassandra
 Data locality for processing
 Scales with the cluster

 More information
 http://cassandra.apache.org
 http://wiki.apache.org/cassandra/HadoopSupport
 Cassandra:The Definitive Guide
 About me:
 jeremy.hanna@rackspace.com
 @jeromatron onTwitter
 jeromatron on IRC in #cassandra

Cassandra + Hadoop @ApacheCon

More Related Content

What's hot (20)

Viewers also liked (12)

Similar to Cassandra + Hadoop @ApacheCon (20)

More from Jeremy Hanna (12)

Recently uploaded (20)

Cassandra + Hadoop @ApacheCon

Editor's Notes