Hybrid MongoDB and RDBMS Applications

Hybrid MongoDB
Applications
with Relational Databases

Today’s Agenda
•Who I am
•Why MongoDB w/intro
•Why Hybrid
•Hybrid Case Studies
•How OpenSky implemented Hybrid
MySQL / MongoDB

My name is
Steve Francia

@spf13

•15+ years building the
internet (13 years using SQL)

•Father, husband,
skateboarder

•Chief Solutions Architect @
10gen responsible for
drivers, integrations, web &
docs

• Company behind MongoDB
• AGPL license, own copyrights, engineering team
• support, consulting, commercial license revenue
• Management
• Google/DoubleClick, Oracle, Apple, NetApp
• Funding: Sequoia, Union Square, Flybridge
• Ofﬁces in NYC, Palo Alto, London & Dublin
• 90+ employees

Before 10gen I
worked
for

http://opensky.com

OpenSky was the ﬁrst
e-commerce site built
on MongoDB

Why MongoDB
My Top 10 Reasons

10. Great developer experience
9. Speaks your language
8. Scale horizontally
7. Fully consistent data w/atomic operations

1.It’ssource scale
web
6. Memory Caching integrated
5. Open
4. Flexible, rich & structured data format not just K:V
3. Ludicrously fast (without going plaid)
2. Simplify infrastructure & application

MongoDB is
Application Document
Oriented
{ author: “steve”,
High date: new Date(),
text: “About MongoDB...”,
Performance tags: [“tech”, “database”]}

Fully
Consistent

Horizontally Scalable

Under the hood

• Written in C++
• Runs on nearly anything
• Data serialized to BSON
• Extensive use of memory-mapped ﬁles
i.e. read-through write-through memory
caching.

Database Landscape
MemCache
Scalability & Performance

MongoDB

RDBMS

Depth of Functionality

This has led
some to say

“
MongoDB has the best
features of key/values
stores, document databases
and relational databases in
one.
John Nunemaker

Reasons to build a
hybrid application
•Friction in existing application caused
by RDBMS
•Transitioning an existing application to
MongoDB
•Using the right tool for the right job
•Need some features not present in
MongoDB

Reasons Not to build
a hybrid application
•Aggregation (at least not very soon)
•Lack of clear understanding of needs
•Backups
•MongoDB as cache in front of SQL
•Loads more...

Hybrid
Applications...
but I don’t
want to
complicate
things

Most
RDMBS
applications
are already
hybrid

Typical RDMBS
Application

Memcache

App

RDBMS

Typical Hybrid
RDMBS Application

MongoDB

App

RDBMS

Most of the same
rules apply

•Application partitions data between
two (or more) systems.
•Model layer tracks what content
resides where.

Hybrid is easier than
RDMBS + MemCache
• Always know where to ﬁnd a piece of data.
• Data never needs expiring.
• Data not duplicated (for the most part)
across systems.
• Always handle a record same way.
• Developer freedom to choose the right tool
for the right reasons.

Typical RDBMS
retrieval operation
exists & up to date?
if yes... then done Memcache

if no, query DB for it
Retrieve record(s) RDBMS
App
Replace in cache
Memcache
Repeat

Typical Hybrid
Retrieval Operation
ﬁnd
return MongoDB

App OR
query
return RDBMS

Typical RDMBS
write operation
insert or update row
conﬁrm written RDBMS

assemble into object(s)
App write object

Memcache

Typical RDMBS
write operation

App write object
write object
write object Memcache
write object
This goes on for a while doesn’t it?

Typical RDMBS
write operation

App write object
write object
write object Memcache
write object
This goes on for a while doesn’t it?

one row can be in many objects so there’s
a lot of complication in updating everything

Typical Hybrid
Write Operation
save document
return MongoDB

App OR
insert or update
return RDBMS

Archiving
Why Hybrid:
• Existing application built on MySQL
• Lots of friction with RDBMS based archive storage
• Needed more scalable archive storage backend
Solution:
• Keep MySQL for active data (100mil), MongoDB for archive (2+
bil)
Results:
• No more alter table statements taking over 2 months to run
• Sharding ﬁxed vertical scale problem
• Very happily looking at other places to use MongoDB

Reporting
Why Hybrid:
• Most of the functionality written in MongoDB
• Reporting team doesn’t want to learn MongoDB

Solution:
• Use MongoDB for active database, replicate to MySQL for
reporting

Results:
• Developers happy
• Business Analysts happy

E-commerce
Why Hybrid:
• Multi-vertical product catalogue impossible to model in RDBMS
• Needed transaction support RDBMS provides

Solution:
• MySQL for orders, MongoDB for everything else

Results:
• Massive simpliﬁcation of code base
• Rapidly build, halving time to market (and cost)
• Eliminated need for external caching system
• 50x+ improvement over MySQL alone

How

implemented a
hybrid MongoDB /
MySQL solution
http://opensky.com

Doctrine (ORM/ODM)
makes it easy

Data to store in SQL

•Order
•Order/Shipment
•Order/Transaction
•Inventory

Data to store in
MongoDB
• User • Event
• Product • TaxRate
• Product/Sellable • ... and then I got
tired of typing them in
• Address
• Just imagine this list
• Cart has 40 more classes

• CreditCard • ...

The most boring SQL
schema ever

CREATE TABLE `product_inventory` (
`product_id` char(32) NOT NULL,
ìnventory` int(11) NOT NULL DEFAULT '0',
PRIMARY KEY (`product_id`)
);

CREATE TABLE `sellable_inventory` (
`sellable_id` char(32) NOT NULL,
ìnventory` int(11) NOT NULL DEFAULT '0',
PRIMARY KEY (`sellable_id`)
);

CREATE TABLE òrders` (
ìd` int(11) NOT NULL AUTO_INCREMENT,
ùserId` char(32) NOT NULL,
`shippingName` varchar(255) DEFAULT NULL,
`shippingAddress1` varchar(255) DEFAULT NULL,
`shippingAddress2` varchar(255) DEFAULT NULL,
`shippingCity` varchar(255) DEFAULT NULL,
`shippingState` varchar(2) DEFAULT NULL,
`shippingZip` varchar(255) DEFAULT NULL,
`billingName` varchar(255) DEFAULT NULL,
`billingAddress1` varchar(255) DEFAULT NULL,
`billingAddress2` varchar(255) DEFAULT NULL,
`billingCity` varchar(255) DEFAULT NULL,

Did you notice
Inventory is in SQL
But it’s also property in your Mongo collections?

Inventory is
transient
• Product::$inventory is effectively a
transient property
•Note how I said “effectively”? ... we
cheat and persist our transient
property to MongoDB as well
•We can do this because we never really
trust the value stored in Mongo

Accuracy is only important
when there’s contention

•For display, sorting and alerts, we can
use the value stashed in MongoDB
•It’s faster
•It’s accurate enough

•For display, sorting and alerts, we can
use the value stashed in MongoDB
•It’s faster
•It’s accurate enough
•For ﬁnancial transactions, we want the
multi table transactions from our
RDBMS.

Inventory kept in
sync with listeners

Inventory kept in
sync with listeners
•Every time a new product is created,
its inventory is inserted in SQL

Inventory kept in
sync with listeners
•Every time an order is placed,
inventory is veriﬁed and decremented

Inventory kept in
sync with listeners
•Every time an order is placed,
inventory is veriﬁed and decremented
•Whenever the SQL inventory changes,
it is saved to MongoDB as well

Be careful what you
lock

Be careful what you
lock
1. Acquire inventory row lock and begin
transaction
2. Check current product inventory
3. Decrement product inventory
4. Write the Order to SQL
5. Update affected MongoDB documents
6. Commit the transaction
7. Release product inventory lock

Making MongoDB
and RDBMS relations
play nice

Products are
documents stored
in MongoDB

/** @mongodb:Document(collection="products") */
class Product
{
/** @mongodb:Id */
private $id;

/** @mongodb:String */
private $title;

public function getId()
{
return $this->id;
}

public function getTitle()
{
return $this->title;
}

public function setTitle($title)
{
$this->title = $title;
}
}

Orders are entities
stored in an RDBMS

/**
* @orm:Entity
* @orm:Table(name="orders")
* @orm:HasLifecycleCallbacks
*/
class Order
{
/**
* @orm:Id @orm:Column(type="integer")
* @orm:GeneratedValue(strategy="AUTO")
*/
private $id;

/**
* @orm:Column(type="string")
*/
private $productId;

/**
* @var DocumentsProduct
*/
private $product;

// ...
}

So how does an
RDBMS have a
reference to something
outside the database?

Setting the Product
class Order {

// ...

public function setProduct(Product $product)
{
$this->productId = $product->getId();
$this->product = $product;
}
}

• $productId is mapped and persisted
• $product which stores the Product
instance is not a persistent entity
property

OrderPostLoadListener
use DoctrineORMEventLifecycleEventArgs;

class OrderPostLoadListener
{
public function postLoad(LifecycleEventArgs $eventArgs)
{
// get the order entity
$order = $eventArgs->getEntity();

// get odm reference to order.product_id
$productId = $order->getProductId();
$product = $this->dm->getReference('MyBundle:DocumentProduct', $productId);

// set the product on the order
$em = $eventArgs->getEntityManager();
$productReflProp = $em->getClassMetadata('MyBundle:EntityOrder')
->reflClass->getProperty('product');
$productReflProp->setAccessible(true);
$productReflProp->setValue($order, $product);
}
}

All Together Now
// Create a new product and order
$product = new Product();
$product->setTitle('Test Product');
$dm->persist($product);
$dm->flush();

$order = new Order();
$order->setProduct($product);
$em->persist($order);
$em->flush();

// Find the order later
$order = $em->find('Order', $order->getId());

// Instance of an uninitialized product proxy
$product = $order->getProduct();

// Initializes proxy and queries the monogodb database
echo "Order Title: " . $product->getTitle();
print_r($order);

Read more about
this technique
Jon Wage, one of OpenSky’s engineers, ﬁrst
wrote about this technique on his personal
blog: http://jwage.com

You can read the full article here:
http://jwage.com/2010/08/25/blending-the-
doctrine-orm-and-mongodb-odm/

http://spf13.com
http://github.com/spf13
@spf13

Questions?
download at mongodb.org
PS: We’re hiring!! Contact us at jobs@10gen.com

Hybrid MongoDB and RDBMS Applications

Hybrid MongoDB and RDBMS Applications

More Related Content

What's hot (20)

Similar to Hybrid MongoDB and RDBMS Applications (20)

More from Steven Francia (20)

Recently uploaded (20)

Hybrid MongoDB and RDBMS Applications

Editor's Notes