No SQL, No problem - using MongoDB in Ruby

no SQL, no problem
development with mongodb and Sinatra

Sam Beam
Onset Corps

A Brief History of
Relational Databases*
(*as if you need it)

• “A Relational Model of Data for
Large Shared Databanks” - Edgar
F. Codd, 1970 -- IBM

• “Standard” SQL - Structured
Query Language

• ACID (atomicity, consistency,
isolation, durability)
photo: osti.gov

The Relational Data Model

• An engine based on “rules” and “facts”
• Consistency/Isolation self-enforced
• ACID

SQL

• History
• Purpose
• “Standards”
• Extensions

SQL

• History
• 1974 - IBM Research “SEQUEL”
Ingres, UC Berkley
• 1979 - 1983 First IBM releases

SQL

• Purpose
SELECT name FROM emp
WHERE salary > 55000
AND dept = ’sales’

• Simple, set-based, declarative syntax brochure: 1980-85 computerhistory.org

DEC microcomputer 1983 (128kB RAM)
computerweekly.com

SQL
IBM Starlink Workstations, 1983
computerweekly.com

• “Standards”
• ANSI SQL-92

• no existing RDBMS in full compliance

SQL IBM AS/400
computerweekly.com

• Extensions
• Procedural languages (PL/SQL, T-SQL,pgSQL etc)

• Storage type extensions (BLOB, XML, Java)

SQL Extensions
SQL/XML SELECT XMLElement
(name emp,
Store documents as XMLForest(last_name || ’,’ ||
CLOB first_name AS fullname, salary) )
FROM emp;
or mapped to columns

Tools for <emp>
‣Annotating <fullname>Tiger, Scott</fullname>
‣Indexing <salary>10000</salary>
‣Searching (XPath) </emp>
‣Validating (DTD) <emp>
<fullname>Smith, John</fullname>
<salary>12000</salary>
XML documents directly </emp>

SQL Extensions

Example of specialized, sparse data

Geospatial/Vector
• Vector Markup Language (VML)
Several choices - all XML • Scalable Vector Graphics (SVG)
• Geography Markup Language (GML) • LandXML
• Keyhole Markup Language • X3D
• G P S eXchange Format (GP X ) • VRML

SQL Extensions
SELECT TO_NUMBER(EXTRACTVALUE(VALUE(t1), 'trk/number')) as track_number,
SUBSTR(EXTRACTVALUE(VALUE(t1), 'trk/name'),1,10) as track_name,

Example of specialized, sparse data
mdsys.sdo_geometry(CASE WHEN (SELECT COUNT(*)
FROM TABLE(XMLSequence(EXTRACT(VALUE(allSegments),
'trk/trkseg',
'xmlns="http://www.topografix.com/GPX/1/1"')))) > 1
THEN 3006
ELSE 3002

Geospatial/Vector END,
8307,
NULL, • Vector Markup Language (VML)
Several choices - all XML
GetElemInfoFromXML(EXTRACT(VALUE(allSegments),'trk/trkseg','xmlns="http://www.topografix.com/GPX/1/1"')),
• Scalable Vector Graphics (SVG)
• Geography Markup Language (GML)
CAST(MULTISET(SELECT case when mod(rownum,3) = 1 then TO_NUMBER(EXTRACT(VALUE(t), '/trkpt/@lon','xmlns="http://
• LandXML
• Keyhole Markup Language when mod(rownum,3) = 2 then TO_NUMBER(EXTRACT(VALUE(t), '/trkpt/@lat','xmlns="http://
• X3D
• G P S eXchange Format (GP X )
when mod(rownum,3) = 0 then TO_NUMBER(EXTRACT(VALUE(t), '/trkpt/ele/text()','xmlns="h
• VRML
1/1"'))
end ordinate
FROM (select level as rin from dual connect by level < 4) r,
TABLE(XMLSequence(EXTRACT(VALUE(allSegments),'trk/trkseg/trkpt','xmlns="http://www.topogr
) as mdsys.sdo_ordinate_array
)) as geom
FROM GPX2 g,
TABLE(XMLSequence(EXTRACT(g.OBJECT_VALUE,'/gpx/trk','xmlns="http://www.topografix.com/GPX/1/1"'))) t1,
TABLE(XMLSequence(EXTRACT(VALUE(t1),'trk[number=' || EXTRACTVALUE(VALUE(t1), 'trk/number') || ']','xmlns="http://
1/1"'))) allSegments;

Typical RDBMS Schema
Issues

• Conﬂicts
• Downtime
• Deployment
• Scaling
image credit: www.magentocommerce.com

Typical RDBMS Schema
“For purposes of flexibility, the Magento
Issues database heavily utilizes an Entity-Attribute-
Value (EAV) data model. As is often the case,
the cost of flexibility is complexity -
Magento is no exception. The process of

• Conflictsmanipulating data in Magento is often more
“involved” than that typically experienced using

• Downtime
traditional relational tables.”
http://www.magentocommerce.com/wiki/

• Deployment
development/

• Scaling
image credit: www.magentocommerce.com

Entity-Attribute-Value
When you have
Unknown Unknowns

A “thing” and “properties” engine
(instead of “rules” and “facts”)

Often found in:
•e-commerce
•medical records
•event logging
•science image: Yale Univ. School of Medicine
senselab.med.yale.edu

Entity-Attribute-Value

Only alternative:
create table customer(
Cust_ID number,
Cust_Name varchar2(xxx),
Cust_Contact varchar2(xxxx),
field1 varchar2(4000),
field2 varchar2(4000),
...
fieldN varchar2(4000)
)
credit: asktom.oracle.com

Logical Progression
<sarcasm>

new datatype “xml_universe”
XML serialized object
DTD containing all the known attributes of
"everything"

Schema:
CREATE TABLE everything (
xml_universe NOT NULL
);

</sarcasm>

Enter the Dragon

“NoSQL”

Enter the Dragon

“NoSQL”
NoSQL is a movement promoting a
loosely deﬁned class of non-relational
data stores that break with a long history
of relational databases.
-- Wikipedia

“NoSQL”
NoSQL is a movement promoting a
loosely deﬁned class of non-relational
data stores that break with a long history
of relational databases.
-- Wikipedia

• Many techniques
• Many weapons
• Many use cases

“NoSQL”

“Not Only SQL”

“Non-Relational Database
Management System”

“NoSQL”

neg
“Not Only SQL”

ati
ve
“Non-Relational Database
Management System”

Structured Storage
Key-Value Store

Document Store

Multivalue Database

OODBMS

Paradigm Change
CAP Theorem
“One can only have two of Consistency, Availability, and
tolerance to network Partitions at the same time”

Paradigm Change
CAP Theorem

If the network is broken, your database won’t work.

Paradigm Change
CAP Theorem

If the network is broken, your database won’t work.

The network is going to break.

Paradigm Change
CAP Theorem
Consistency

Availability

Paradigm Change

• Non-relational
• Schema-free
• “Easily” scalable
• OO friendly (no ORM)
• web friendly - REST/JSON API

Paradigm Change
When?

• High-volume, low value
(social media, web)
• Unknown Unknowns
• Storage of application objects
• Caching, Logging

Paradigm Change
When NOT?

• ACID
• Traditional Waterfall development
• Problems requiring Relational model *

* combination?

Paradigm Change
Who’s driving it?

The Candidates
Storage Type License Implementation

Cassandra ColumnFamily * Apache 2,0 Java

CouchDB Document Apache 2,0 Erlang

Hbase ColumnFamily * Apache 2,0 Java
Redis Key/Value BSD C

Tokyo Cabinet Key/Value LGPL C
Voldemort Key/Value Apache 2,0 Java

Memcached Key/Value BSD C

MongoDB Document (BSON) AGPL 3.0 C++

No SQL, No problem - using MongoDB in Ruby

• Document-oriented
• Dynamic queries
• Full dynamic index support
• Efﬁcient binary large-object storage
• Built for speed
• Replication and Auto-failover

Installation

‣ Download source or binary for OS X, Linux, Windows
http://www.mongodb.org/

‣ Make data directory
$ mkdir /some/path/mongodb

‣ Run!
$ bin/mongod --dbpath=/some/path/mongodb

Database Structure

‣Separate DBs
‣Organized into Collections
top-level key Document

Collections

‣Group things into logical classes
‣Indexable by one or more keys
‣Schema-free!

Documents

‣Always contains key _id
‣Creating Relationships:
subdocument, shared key, or DBRef

‣Native storage and transfer : BSON

JSON

A collection of name/value pairs.
[object, record, struct, dictionary, hash table, keyed list, associative array]

An ordered list of values.
[array, vector, list, sequence]

http://json.org/

BSON

BSON is a binary encoded serialization
of JSON-like documents.

http://bsonspec.org/

http://www.mongodb.org/display/DOCS/BSON

JSON/BSON Example

{
author : "Joe Example",
created : Date(’03-28-2010’),
title : "My latest blog post",
tags : [ "example", "joe", "testing"],
comments : [
{ author : 'jim', comment : 'I disagree' },
{ author : 'nancy', comment : 'Good post' }
]
}

http://bsonspec.org/

http://www.mongodb.org/display/DOCS/BSON

mongo shell
$ mongo
MongoDB shell version: 1.5.0-pre-
url: test
connecting to: test
type "help" for help
> show dbs
admin
shorty
test
> use test
switched to db test
>

mongo shell
$ mongo
MongoDB shell version: 1.5.0-pre-
url: test
connecting to: test
type "help" for help
> show dbs
admin
shorty
test
> use test
switched to db test
> show collections
foo
fs.chunks
fs.files
system.indexes
>

mongo shell
> db.foo.find()
> db.foo.save( { name : 'Fred Flintstone', catchphrase : 'Yabba Dabba doo' } )
> db.foo.find()
{ "_id" : ObjectId("4bcb1dc899d3ae6c0c68035b"), "name" : "Fred Flintstone",
"catchphrase" : "Yabba Dabba doo" }
>

mongo shell
> db.foo.save( { name : 'Fred Flintstone', catchphrase : 'Yabba Dabba doo' } )
> db.foo.save( { name : 'Barney Rubble' } )
> db.foo.find()
{ "_id" : ObjectId("4bcb213199d3ae6c0c680361"), "name" : "Fred Flintstone",
"catchphrase" : "Yabba Dabba doo" }
{ "_id" : ObjectId("4bcb213799d3ae6c0c680362"), "name" : "Barney Rubble" }
>

mongo shell
> for( var i = 1; i < 10; i++ ) db.things.save( { x:'thing'+i, counter:i } );
>

mongo shell
> for( var i = 1; i < 10; i++ ) db.things.save( { x:'thing'+i, counter:i } );
> db.things.find()
{ "_id" : ObjectId("4bcb22e199d3ae6c0c680363"), "x" : "thing1", "counter" : 1 }
{ "_id" : ObjectId("4bcb22e199d3ae6c0c68036a"), "x" : "thing8", "counter" : 8 }
{ "_id" : ObjectId("4bcb22e199d3ae6c0c68036b"), "x" : "thing9", "counter" : 9 }
>

mongo shell
> db.things.find( { x : 'thing7' } )
>

mongo shell
> db.things.find( { counter : { $gt : 5, $lte : 8 } } )
{ "_id" : ObjectId("4bcb22e199d3ae6c0c68036a"), "x" : "thing8", "counter" : 8 }
>

mongo shell
> db.foo.update( { name : 'Fred Flintstone' }, { $set : { wife : 'Wilma' }} )
> db.foo.update( { name : 'Barney Rubble' }, { $set : { wife : 'Betty' }} )
> db.foo.find()
{ "_id" : ObjectId("4bcb213199d3ae6c0c680361"), "catchphrase" : "Yabba Dabba
doo", "name" : "Fred Flintstone", "wife" : "Wilma" }
{ "_id" : ObjectId("4bcb213799d3ae6c0c680362"), "name" : "Barney Rubble",
"wife" : "Betty" }

mongo shell
> db.foo.update( { name : 'Fred Flintstone' },
{ $set : { kids : ['Pebbles'] } } )

> db.foo.findOne( {name : 'Fred Flintstone'} )
{
"_id" : ObjectId("4bcb213199d3ae6c0c680361"),
"catchphrase" : "Yabba Dabba doo",
"kids" : [
"Pebbles"
],
"name" : "Fred Flintstone",
"wife" : "Wilma"
}

mongo shell
> db.foo.update( { name : 'Fred Flintstone' },
{ $push : { kids : 'Bam-Bam' } } )

> db.foo.findOne( {"_id" : ObjectId("4bcb213199d3ae6c0c680361")})
{
"kids" : [
"Pebbles",
"Bam-Bam"
],
"wife" : "Wilma"
}
>

mongo shell
> thing = db.things.findOne( { counter : 3 } )
{
"_id" : ObjectId("4bcb22e199d3ae6c0c680365"),
"x" : "thing3",
"counter" : 3
}
> fred = db.foo.findOne( {name : 'Fred Flintstone'} )
{
"kids" : [
"Pebbles",
"Bam-Bam"
],
"things" : null,
"wife" : "Wilma"
}
>

mongo shell
> fred.things = []
[ ]
> fred.things.push ( new DBRef('things', thing._id) )
> db.foo.save(fred)
> fred
{
"kids" : [
"Pebbles",
"Bam-Bam"
],
"things" : [
{
"$ref" : "things",
"$id" : ObjectId("4bcb22e199d3ae6c0c680365")
}
],
"wife" : "Wilma"
}

mongo shell
> fred.things[0]
{ "$ref" : "things", "$id" : ObjectId("4bcb22e199d3ae6c0c680365") }
> fred.things[0].fetch()
{
"_id" : ObjectId("4bcb22e199d3ae6c0c680365"),
"x" : "thing3",
"counter" : 3
}
>

and Ruby

mongo-ruby-driver

gem install mongo
gem install mongo_ext

and Ruby

Querying
DB = Connection.new('localhost').db('test')

coll = DB['mycollection']

coll.find( { :first_name => 'Fred' } )     # find all Freds

coll.find( { :email => /@gmail.com$/i } )  # Regex

coll.find_one( { :_id => 42 } )            # single record

coll.find( { :age => { '$gte' => 21 } } )   # native conditionals

coll.find( { 'author.first_name' => 'John' } )    # embedded object

coll.find( { '$where' => 'this.age % 7 == 0' } )  # custom conditional

and Ruby

More Querying
Native conditionals: $in, $nin, $all, $size, $exists, $type

:ﬁelds (subset of document keys)
coll.find({:zipcode => '03801'}, {:fields => [:first_name,:last_name]} )

:limit, :skip for pagination
coll.find({:zipcode => '03801'}, {:limit => 100})

sorting
coll.find({:zipcode => '03801'}, {:sort => [:created_at, 'ascending']})

count, distinct and group (does not use Map/Reduce)

and Ruby

Inserting and ﬁnding
require 'rubygems'
require 'mongo'
include Mongo

DB = Connection.new(ENV['DATABASE_URL'] || 'localhost').db('test')

coll = DB['eyes']

coll.remove

100.times { |i| coll.insert('i' => i, 'token' => rand(1000)) }

coll.find({'token' => { '$gt' => 900 }},
{:sort => [:token, :descending]}).each { |row| puts row.inspect }

and Ruby

Updating in place
coll.update({:token => { '$gt' => 970 }},
{'$set'=>{:winner => true}},
{:multi => true});

coll.find(:winner => true).each { |row| puts row.inspect }

{"_id"=>ObjectID('4bcb51a65ea7db36fc000036'), "i"=>53, "token"=>990, "winner"=>true}
{"_id"=>ObjectID('4bcb51a65ea7db36fc000038'), "i"=>75, "token"=>977, "winner"=>true}

and Ruby

Map/Reduce

Parallel Computing

Delayed Gratiﬁcation

and Ruby

Map/Reduce

Map: chops massive data set into smaller
problem-speciﬁc set

Reduce: iterates over Map results, combining
as needed

and Ruby

Map/Reduce
map    = "function() { emit(this.author, {votes: this.votes}); }"
reduce = "function(key, values) {
  var sum = 0;
  values.forEach(function(doc) {
   sum += doc.votes;
  });
  return {votes: sum};
};"

@results = @comments.map_reduce(map, reduce)

puts @results.find().inspect

[{"author"="sbeam", "value"=>{"votes"=>21.0}},
{"author"=>"barney", "value"=>{"votes"=>13.0}}]

http://kylebanker.com/blog/2009/12/mongodb-map-reduce-basics/

and Ruby

GridFS

Store large ﬁles

Transparently chunks

Incremental delivery (video streaming)

and Ruby

GridFS
@grid = Grid.new(@db)
# Saving IO data and including the optional filename
image = File.open("kitty.jpg")
file_id = @grid.put(image, :filename => "kitty.jpg")

....

@grid = Grid.new(@db)
# writing file with given _id to HTTP
if img = @grid.get(Mongo::ObjectID::from_string(params[:file_id]))
headers 'Content-Type' => img.content_type
img.read
end

and Ruby

Other interesting things
Capped collections (think memcached)

Multikeys and Full-text search

Auto-sharding

Replica Sets

and Ruby

Eventual Consistency

can my use case tolerate
• stale reads?
• reading values out of order?
• not reading my own writes?

and Ruby

Ruby Adapters

• MongoRecord
http://github.com/mongodb/mongo-record
• MongoMapper
http://github.com/jnunemaker/mongomapper
• Candy
http://github.com/SFEley/candy

and Ruby

MongoMapper example
class Post
  include MongoMapper::Document

  key :title, String
  key :body, String

  many :comments, :as => :commentable, :class_name => 'PostComment'
  timestamps!
end

class PostComment
  include MongoMapper::Document

  key :username, String, :default => 'Anonymous'
  key :body, String

  key :commentable_id, ObjectId
  key :commentable_type, String
  belongs_to :commentable, :polymorphic => true

  timestamps!
end

Much more...

• Theory
http://blog.mongodb.org/
• GUIs
http://blog.timgourley.com/tagged/nosql
• Support
http://www.10gen.com/

• sbeam@onsetcorps.net
• http://twitter.com/sbeam
• http://github.com/sbeam/mf1

No SQL, No problem - using MongoDB in Ruby

More Related Content

What's hot (20)

Similar to No SQL, No problem - using MongoDB in Ruby (20)

Recently uploaded (20)

No SQL, No problem - using MongoDB in Ruby

Editor's Notes