Building node.js applications with Database Jones

Copyright © 2013, Oracle and/or its affiliates. All rights
reserved.
Database Jones
! J.D. Duncan, john.duncan@oracle.com
! Craig Russell, craig.russell@oracle.com
1
1

reserved.
J.D.
! Senior Software Engineer, MySQL Cluster, at Oracle
! Former Unix Sysadmin & web developer
! MySQL AB (2004) - Sun (2008) - Oracle (2011)
! Projects
! Database Jones, NDB Memcache, mod_ndb
2
2

reserved.
Craig
! Architect, Oracle Corp.
! Java Data Objects JSR-243 Specification Lead
! Projects
! Database Jones connector for MySQL Cluster
! Cluster/J Java Connector for MySQL Cluster
! Secretary, Apache Software Foundation
3
3

Why Database Jones?
•Node.JS
• Highly scalable Javascript platform for web services
•MySQL Cluster
• Highly scalable database engine
•Database Jones
• Highly scalable API for database access from Node.JS
4

MySQL Cluster Architecture
MySQL C
6

Database Jones
•Fast, easy database API for Node.js
•JavaScript for user API and common operations
•Two adapters currently shipping
• NDB: Native adapter to MySQL Cluster C++ API
• MySQL: Adapter to node-mysql (third party tool)
• Open architecture allows other adapters
•Most operations are asynchronous
7

Data Model
•Two models are supported:
• Operations using table names return plain objects
• {"id": 554, "first_name": "Roy",
"last_name": "Raye"}
• Operations using JavaScript constructors return instances
of JavaScript classes
8

Object Mapping
•Table mapping defines the relationship between
• Tables and columns
• Objects and fields
• Foreign keys and object relationships
•Table mapping is flexible
• Change field names
• Provide transformations of data
• Database format to JavaScript format
• e.g. Date types, True/False, custom formats
9

Mapped Table Formats
•Classic
• Fields are stored in individual columns
•Serialized
• Objects are serialized into JSON and stored in a JSON
column
•Hybrid
• Some ﬁelds are stored in their own column
• All other ﬁelds are serialized
10

Classic Format
•Most appropriate for existing tables/schema
•Schema is deﬁned outside the application
•Compose complex objects via database joins
11

Serialized
•Most appropriate for “schema-less” designs
•Limit query capability (we’re working on it)
•Limited join capability (we’re working on this too)
12

Hybrid
•Combines best features of serialized and classic
•Some fields are stored in their own column
• object composition via joins
•All other fields are stored serialized
• Object composition via serialization
•Hybrid tables can store arbitrarily complex objects
efficiently
13

Database Jones Operations
•Metadata (non-transactional)
• List tables
• Get metadata for a table
•CRUD (transactional)
• Insert (save)
• Find by key (includes joins)
• Delete by key
• Update by key
•Query (transactional)
• Complex query criteria
14

reserved.
Install & Demo
15

Install from Github
✦ git clone http://github.com/mysql/mysql-js
✦ git config --global --add core.symlinks true
github.com/mysql/mysql-js
You may need to allow git to make symlinks:
16

Components
The user API
DB Service Providers
Network Conﬁguration
A directory of symlinks
Sample Application code
Uniﬁed debugging
library for C++ and JS
Benchmark
17

Architecture
Jones
Application Code
DB Service Provider
Jones API
Jones SPI
Database
Database Network Protocol
18

Using the DB Service Providers
•MySQL
• npm install mysql
•NDB
• cd jones-ndb
• node configure
19

Sample Twitter-like Application
•In samples/tweet
•5 tables
20

Contents of samples/tweet
4 simple API demos
2 SQL scripts
Main Application
(Command Line & HTTP)
Demos of tweet.js
21

tweet.js
Usage: node tweet {options} {command} {command arguments}
-a <adapter>: run using the named adapter (default: ndb)
-h or --help: print this message
-d or --debug: set the debug flag
--detail: set the detail debug flag
-df <file>: enable debug output from <file>
-E or --deployment <name>: use deployment <name> (default: te
COMMANDS:
put user <user_name> << JSON Extra Fields >>
get user <user_name>
delete user <user_name>
post tweet <author> << Message >>
get tweet <tweet_id>
delete tweet <tweet_id>
put follow <user_follower> <user_followed>
get followers <user_name>
get following <user_name>
get tweets-by <user_name>
get tweets-at <user_name>
get tweets-about <hashtag>
get tweets-recent <count>
start server <server_port_number>
22

Author table: Hybrid data model
CREATE TABLE author (
user_name varchar(20) CHARACTER SET UTF16LE not null,
full_name varchar(250),
tweet_count int unsigned not null default 0,
SPARSE_FIELDS varchar(4000) CHARACTER SET utf8,
PRIMARY KEY(user_name)
) ;
“Catch All” JSON Column
from create_tweet_tables.sql
23

Hybrid Insert into Author
node tweet put user caligula
'{ "full_name": "Gaius Julius Caesar Germanicus" ,
"profile_text": "I am your little boot!" }'
user_name full_name
SPARSE_FIELDS
from demo_populate_data.sh
24

API Essentials
•SessionFactory is a heavyweight master connection
•Session is a lightweight pooled connection for a user
• e.g. for an HTTP request
• a session has a single active transaction
•Session provides APIs for database operations
• Example: use session.ﬁnd() to ﬁnd a single row of data
25

API Sample application: find.js
/* This script shows an example find() operation using a table
name and primary key, and working with promises.
For a similar example using callbacks rather than promises,
see insert.js
*/
"use strict";
var jones = require("database-jones");
26

find.js: Configure a database
connection
/* new ConnectionProperties(adapter, deployment)
The first argument names a database backend, e.g. "ndb",
"mysql", etc.
The second argument names a "deployment" defined in a
jones_deployments.js file. (A default file can be found
two directories up from here). jones_deployments.js is the
preferred place to customize the host, username,
password, and other parameters of the database connection.
*/
var connectionProperties =
new jones.ConnectionProperties("mysql", "test");
27

find.js: Process the command line
/* node find.js table_name primary_key_value
argv[0] argv[1] argv[2] argv[3] */
if (process.argv.length !== 4) {
console.log("Usage: node find <table> <key>n");
process.exit(1);
}
var table_name = process.argv[2],
find_key = process.argv[3];
28

find.js: Run the database operation
/* This version of openSession() takes one argument and returns a
promise. The argument is the set of connection properties
obtained above.
Once the session is open, use it to find an object.
find() is a Jones API call that takes a primary key or unique
key and, on success, returns *only one object*.
*/
jones.openSession(connectionProperties).
then(function(session) {
return session.find(table_name, find_key);
}).
then(console.log, console.trace). // log the result or error
then(jones.closeAllOpenSessionFactories); // disconnect
29

Many varieties of find()
The first argument can be:
• Table Name
• JavaScript constructor that has been mapped
• Projection describing a structure of related objects
30

The second argument can be:
• String or Number
1-part primary key lookup
• { key_field_name : value }
Primary key or any unique index
• { key_part_1 : value1, key_part_2: value2 }
Multi-part primary key or unique index
31

• The optional third argument can be a callback
• Any extra arguments after the third will be supplied
to the callback (for additional context)
• find() always returns a promise
(even if you don’t use the promise)
32

insert.js: Callback Style
/* This script shows an example persist() operation using a table
name and primary key, and working with callbacks.
*/
function disconnectAndExit(status) {
jones.closeAllOpenSessionFactories(function() {
process.exit(status);
});
}
/* handleError() exits if "error" is set,
or otherwise simply returns.
*/
function handleError(error) {
if(error) {
console.trace(error);
disconnectAndExit(1);
}
}
33

insert.js: Run the operation
/* This version of openSession() takes three arguments:
ConnectionProperties
A table name, which will be validated upon connecting
A callback which will receive (error, session)
*/
jones.openSession(connectionProperties, table_name,
function(err, session) {
handleError(err);
/* The callback for persist() only gets one argument */
session.persist(table_name, object, function(err) {
handleError(err);
console.log("Inserted: ", object);
disconnectAndExit(0);
});
});
34

scan.js: Query
/* This script provides an example of the Jones Query API.
In this example, we query the tweet table for posts
by a particular author, and apply a sort and limit
on the query.
*/
"use strict";
if (process.argv.length < 3 ) {
console.log("usage: node scan <author> [limit] [order]");
process.exit(1);
}
35

scan.js: Query Parameters
// node scan.js <author> [limit] [order]
var connectionProperties =
new jones.ConnectionProperties("ndb", "test"),
queryTerm = process.argv[2],
limit = Number(process.argv[3]) || 20,
order = (process.argv[4] == "asc" ? "asc" : "desc");
36

scan.js: Query Operation
jones.openSession(connectionProperties).
return session.createQuery("tweet");
}).
then(function(query) {
/* Here we can define query conditions.
For more details see API-documentation/Query
*/
query.where(query.author_user_name.eq(queryTerm));
/* Then execute the query, using limit & order parameters.
*/
return query.execute({ "limit" : limit, "order" : order });
}).
then(jones.closeAllOpenSessionFactories); // disconnect
37

Session Methods
✦find()
✦remove()
✦persist()
✦update()
✦save()
✦load()
Metadata Operations
Key Operations
✦listTables()
✦getTableMetadata()
Query Operations
✦createQuery()
Others
✦currentTransaction()
✦close()
✦createBatch()
✦setPartitionKey()
38

Batch: powerful grouping of
“mixed” key operations
batch = session.createBatch();
tag = tags.hash.pop(); // # hashtags
while(tag !== undefined) {
tagEntry = new HashtagEntry(tag, tweet);
batch.persist(tagEntry);
tag = tags.hash.pop();
}
return batch.execute();
✦batch.find()
✦batch.remove()
✦batch.persist()
✦batch.update()
✦batch.save()
✦batch.load()
39

Example from tweet.js
/* Insert a tweet.
- Start a transaction.
- Persist the tweet & get its auto-increment id.
- Create & persist #hashtag & @mention records
(all in a single batch).
- Increment the author's tweet count.
- Then commit the transaction.
*/
function InsertTweetOperation(params, data) {
[ ... ]
session.currentTransaction().begin();
session.persist(tweet).
then(function() { return session.find(Author, authorName);}).
then(incrementTweetCount).
then(createTagEntries).
then(commitOnSuccess, rollbackOnError).
then(function() {return tweet;}).
then(this.setResult).
then(this.onComplete, this.onError);
40

reserved.
Query & Projection APIs
41

Query API
•Session is Query factory using asynchronous api
•Query has a ﬁlter based on the mapped object
•Filter has comparators
• eq, ne, gt, ge, le, lt, between, isNull, isNotNull
•Filter has boolean operations
• and, or, not, andNot, orNot
•Query execution is asynchronous
•Filter determines query strategy
• Primary/unique key lookup; index scan; table scan
•Properties govern query execution
•Results are given in callback
42

Query Comparators
•Comparators compare properties to parameters
•Query Domain Type property names correspond to
Constructor ﬁeld names (properties)
•Parameters are created by name
• qdt.param('date_low')
•Properties are referenced by ﬁeld name in Constructor
• qdt.date_created
•Comparators are properties of qdt properties
• qdt.date_created.gt(qdt.param('date_low'));
•Comparators return predicates
43

Query Operators
•Predicates are used as query ﬁlters via where function
•Predicates are results of comparators or operators
•Operators combine predicates: and, or, andNot, orNot, not
• var predicate1 = qdt.date_created.gt(qdt.param('date_low'));
• var predicate2 = qdt.date_created.lt(qdt.param('date_high'));
• var predicate = predicate1.and(predicate2);
• var predicate = predicate3.andNot(predicate4);
• qdt.where(predicate)
44

Query Execution
•var promise = query.execute(parameters,
callback);
•parameters: regular javascript object
• skip: number of results to skip over
• limit: number of results to return
• order: 'asc' or 'desc'
• user-speciﬁed parameters deﬁned by q.param
•callback(err, results)
•results: result[ ], possibly no elements
45

Query Example
session.createQuery(Employee).
then(function(query) {
return query.where(query.age.gt(50).and(query.salary.lt(50000))).
execute({limit: 20})}).
then(function(results) {
results.forEach(function(result) {
console.log('Name:', result.name, 'age:', result.age);
})
}).
then(closeSession, reportError);
46

Projection
•Composition of complex objects by joins
• one to one, one to many, many to one, many to many
•Define relationships (bidirectional) in table mapping
•Choose fields to select in Projection
•Choose relationships to join in Projection
•find() can take a Projection as the first argument
47

Projection API
•Projection constructor deﬁnes the domain object
•Fields are added to Projection
•Relationships to other Projections are added to Projection
•Loops are not supported
•Arbitrary nesting depth
48

join.js
"use strict";
/* Constructors for application objects */
function Author() { }
function Tweet() { }
49

join.js: TableMapping for Author
/* TableMappings describe the structure of the data. */
var authorMapping = new jones.TableMapping("author");
authorMapping.applyToClass(Author);
authorMapping.mapSparseFields("SPARSE_FIELDS");
authorMapping.mapOneToMany(
{ fieldName: "tweets", // field in the Author object
target: Tweet, // mapped constructor
targetField: "author" // target join field
});
50

join.js: TableMapping for Tweet
/* TableMappings describe the structure of the data. */
var tweetMapping = new jones.TableMapping("tweet");
tweetMapping.applyToClass(Tweet);
tweetMapping.mapManyToOne(
{ fieldName: "author", // field in the Tweet object
target: Author, // mapped constructor
foreignKey: "author_fk" // SQL foreign key relationship
});
51

join.js: Projection
/*
Projections describe the structure to be returned from find().
*/
var tweetProjection = new jones.Projection(Tweet);
tweetProjection.addFields(["id","message","date_created"]);
var authorProjection = new jones.Projection(Author);
authorProjection.addRelationship("tweets", tweetProjection);
authorProjection.addFields(["user_name","full_name"]);
52

join.js: Run the database operation
/* The rest of this example looks like find.js,
only using find() with a projection rather than a table name.
*/
jones.openSession(new jones.ConnectionProperties("mysql", "test")).
return session.find(authorProjection, find_key);
}).
then(jones.closeAllOpenSessionFactories). // disconnect
then(process.exit, console.trace);
53

join.js: Result
$ node join.js nero
{ user_name: 'nero',
full_name: 'Lucius Domitius Ahenobarus',
tweets:
[ { id: 3,
message: 'I love to sing!',
date_created: Thu Oct 01 2015 16:09:46 GMT-0700 (PDT) },
{ id: 4,
message: 'I am the best #poet and the best #gladiator!',
date_created: Thu Oct 01 2015 16:09:46 GMT-0700 (PDT) } ] }
54

reserved.
Thank you Code Camp!
✦ git clone http://github.com/mysql/mysql-js
55

Building node.js applications with Database Jones

More Related Content

Viewers also liked (6)

Similar to Building node.js applications with Database Jones (20)

Recently uploaded (20)

Building node.js applications with Database Jones