MySQL Document Store and Node.JS

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. |
Reggie Burnett (presenting)
Rui Quelhas (original author of most content)
Monday October 29, 2018
MySQL Document Store and Node.JS

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Confidential – Oracle Internal/Restricted/Highly Restricted
Safe Harbor Statement
The following is intended to outline our general product direction. It is intended
for information purposes only, and may not be incorporated into any contract. It
is not a commitment to deliver any material, code, or functionality, and should
not be relied upon in making purchasing decisions. The development, release,
timing, and pricing of any features or functionality described for Oracle’s
products may change and remains at the sole discretion of Oracle Corporation.

Who Am I?
4
Reggie Burnett
Manager - MySQL Connectors and Plugins
reggie.burnett@oracle.com
Many thanks to Rui Quelhas for the slides and all his great work on the
Node.JS driver! Rui’s original slides should be on slideshare very soon!

Why Document Store?
• Some data shapes simply do not fit well into tables
• Anyone ever use Microsoft Commerce Server?
• Using a single database technology makes:
• Support simpler
• Maintenance simpler
• Scale-out/scale-up simpler
• We can leverage table data and document data in the same
queries!

Why a new API?
• We wanted to mix the power of MySQL’s powerful relational engine
with our new Document Store
• We *didn’t* want to be required to pull our table data into a new
structure
• We wanted to write simpler and easier to read API code
• We wanted an API that had common core elements across all
connectors.
The result is the MySQL X DevAPI!

X DevAPI
•High-level database API to develop modern applications powered by  
InnoDB Cluster [TIP3065, HOL2988]
•Application gateway for the underlying document-store infrastructure (X  
Protocol, Router, X Plugin)
•Off-the-shelf support for CRUD NoSQL document operations
•Lower-level management via raw SQL and other advanced features
•Standard specification for all the official client implementations
•Available in connectors for the most popular languages as well as the
brand new MySQL Shell

What Connectors Support DevAPI
• .NET
• Java
• Python
• Javascript (Node)
• C++
• PHP
• C
Internal Connectors
• Ruby
• Go
• Perl
• Dart/Swift??
Working to bring DevAPI to the
following external connectors

MySQL Document Store
• It’s now possible to save and retrieve unstructured data using a MySQL
database (not talking about raw JSON)
• Data lives in a JSON column but everything is abstracted way from the
user, which only deals with documents
• No need to worry about schemas and data types
• Keeping logical consistency and ACID (it’s MySQL after all)
• At the same time, a schema can still mix in traditional tables
• ...and more https://lefred.be/content/top-10-reasons-for-nosql-with-
mysql/

Component Overview
Client App
MySQL
Router
X DevAPI X Plugin
X Protocol
https://dev.mysql.com/doc/refman/8.0/en/document-store.html
Classic Protocol
10

MySQL Router
• MySQL Router is an integral part of InnoDB Cluster
• Lightweight middleware that provides transparent routing between
an application and back-end MySQL servers
• Can be used for a wide variety of use cases, but mainly to address
high availability and scalability concerns
• Runs as a standalone component
• Packaged with the MySQL server for the usual platforms
• Documentation available at https://dev.mysql.com/doc/mysql-
router/8.0/en/

MySQL X Plugin
• New MySQL server plugin (developed internally)
• Installed and enabled by default on MySQL 8.0 series
• Provides the document store infrastructure on top of plain MySQL
servers
• Uses a different set of ports (33060 instead of the usual MySQL 3306)
• Implements a client-server protocol based on Google Protocol Buffers
• Accepts X DevAPI client requests and processes them using the core
MySQL engine
• Documentation available at https://dev.mysql.com/doc/refman/8.0/
en/document-store.html

MySQL X Protocol
• New client-server protocol based on Google Protocol Buffers (easier for
ANYONE to extend and improve)
• Content awareness
• CRUD expression trees to establish operation boundaries
• Expectations to define conditions and create pipelined statements
• Placeholder assignment and value escaping
• Security baked in
• SSL/TLS by default
• No information leaked to unauthenticated users (e.g. server version)
• Open source at https://dev.mysql.com/doc/internals/en/x-protocol.html

X Protocol
Message example
Mysqlx.Crud.Find {
  collection { name: "collection_name", schema: "schema_name" }
  data_model: DOCUMENT
  criteria {
    type: OPERATOR
    operator {
      name: "=="
      param {
        type: IDENT,
        identifier { name: "_id" }
      }
      param {
        type: LITERAL,
        literal {
          type: V_STRING,
          v_string: { value: "some_string" }
        }
      }
    }
  }
}
 
14

Documents and Collections
• Documents in the MySQL Document store are just JSON
– “JavaScript Object Notation”
– Cross-platform serialization format (common for web services and
applications)
– Standardized as ECMA-404 (http://json.org)
– Supported by a proper native MySQL data type
• Multiple documents are stored inside a Collection
– Technically, an InnoDB table
– One regular column of type JSON
– Virtual columns on top for indexes
15

What does it look like?
SHOW CREATE TABLE
CREATE TABLE `docStoreCollection` (
`doc` json DEFAULT NULL,
`_id` varbinary(32) GENERATED ALWAYS AS
(json_unquote(json_extract(`doc`,_utf8mb4'$._id'))) STORED NOT NULL,
PRIMARY KEY (`_id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci
 
16

What does it look like?
DESCRIBE/SHOW INDEXES
+-------+---------------+------+-----+---------+------------------+
| Field | Type | Null | Key | Default | Extra |
+-------+---------------+------+-----+---------+------------------+
| doc | json | YES | | NULL | |
| _id | varbinary(32) | NO | PRI | NULL | STORED GENERATED |
+-------+---------------+------+-----+---------+------------------+
+------------+-------------+------------+------------+
| INDEX_NAME | COLUMN_NAME | INDEX_TYPE | IS_VISIBLE |
+------------+-------------+------------+------------+
| PRIMARY | _id | BTREE | YES |
+------------+-------------+------------+------------+
17

Diving into the X DevAPI
18

What do I get?
• Fluent API and flexible parameters
• Multiple connection modes (with support for connection pooling)
• Automatic connection failover and timeout management
• Security best practices with multiple SSL/TLS operation modes and
authentication mechanisms as well as improved SQL injection protection
• Small and portable expression-based query language (subset of SQL)
• Hybrid CRUD API for both Document store collections and regular tables
• SQL statements for everything else
• ACID (transactions and row locking)
19

Fluent API
• Code flows from a single point of entry – getSession()
• Code becomes more readable, maintainable (and even testable)
• Operations encapsulated in specialized and semantic methods
• Nice scaffolding for refactor tasks
• First-class support for text-editor (or IDE) hints and auto-
completion
• Smaller SQL injection surface area
• Common standard between different programming environments
20

Flexible Parameters
• Most public API methods work with:
– Multiple individual parameters
– A single array of parameters
– An object with named properties (where it applies)
21
mysqlx.getSession('root@localhost')
mysqlx.getSession({ user: 'root' })
 
collection.add({ name: 'foo' }).add({ name: 'bar' })
collection.add([{ name: 'foo' }, { name: 'bar' }])
 
collection.find('name = :name').bind('name', 'foo')
collection.find('name = :name').bind({ name: 'foo' })
collection.find().fields('foo', 'bar')
collection.find().fields(['foo', 'bar'])

Connection Types
• Each X DevAPI Session matches a single connection (regular mode)
• Pooling mode (X DevAPI Sessions create or re-use existing
connections)
22
Different operation modes
const url = 'mysqlx://root@localhost:33060/defaultSchema'
 
// create a regular standalone connection
mysqlx.getSession(url)
 
const pooling = { enabled: true, maxIdleTime: 500, maxSize: 25, queueTimeout: 500 } 
const client = mysqlx.getClient(url, { pooling })
 
// acquire a connection from the pool (creates one if it does not exist)
client.getSession()

Connection failover and timeout
• Automatic priority-based failover
• Different timeout management for single-host or multi-host settings
23
Pooling and non pooling setups
// single-host
mysqlx.getSession('mysqlx://root@localhost:33060?connect-timeout=1000');
 
// multi-host
// order-based priority
mysqlx.getSession('mysqlx://root@[localhost:33060, localhost:33061]?connect-timeout=1000')
mysqlx.getSession('root@[localhost:33060, localhost:33061] ')
// weight-based priority
mysqlx.getSession('mysqlx://root@[(localhost:33061, priority=99), (localhost:33060, priority=100)]')
mysqlx.getSession('root@[(localhost:33061, priority=99), (localhost:33060, priority=100)]?connect-timeout=1000')

Security Options
• SSL/TLS is enabled by default, for TCP connections, when creating
a session
• Local UNIX socket connections disable it (not needed) to be faster
• Additional server certificate validation
• Options can be overridden on-demand
24
mysqlx.getSession({ ssl: false })
mysqlx.getSession('mysqlx://root@localhost?ssl-mode=DISABLED')
mysqlx.getSession({ ssl: true, sslOptions: { ca: '/path/to/ca.pem' } })
mysqlx.getSession('mysqlx://root@localhost?ssl-ca=(/path/to/ca.pem)')
mysqlx.getSession({ ssl: true, sslOptions: { ca: '/path/to/ca.pem', crl: '/path/to/crl.pem' } })
mysqlx.getSession('mysqlx://root@localhost?ssl-ca=(/path/to/ca.pem)&ssl-crl=(/path/to/crl.pem)')

Authentication Mechanisms
• SHA-1 and SHA-2 password hashing
• Supports authentication mechanisms for multiple server plugins:
– MYSQL41 for mysql_native_password
– PLAIN and SHA256_MEMORY for sha256_password and/or
caching_sha2_password
25
mysqlx.getSession({ user: 'root', auth: 'MYSQL41' })
mysqlx.getSession('mysqlx://root@localhost?auth=MYSQL41')
mysqlx.getSession({ user: 'root', auth: 'PLAIN' })
mysqlx.getSession('mysqlx://root@localhost?auth=PLAIN')
mysqlx.getSession({ user: 'root', auth: 'SHA256_MEMORY' })
mysqlx.getSession('mysqlx://root@localhost?auth=SHA256_MEMORY')

Document Store API
• DDL API to create new collections and list or remove existing ones
• CRUD API to insert, retrieve, update or remove documents in a collection
• Operate on multiple docs at the collection level or on individual docs
• Additional extensions: projections, aggregation, ordering, limits and
skipping results
• Index management API with support for regular and SPATIAL indexes
• Transactional API to begin, rollback or commit transactions as well as
managing intermediate savepoints
• Document field locking API for granular consistency management
26

Schemaless Data
• X DevAPI introduces a modern CRUD API to work with schemaless
data
• Semantic Document Store methods that allow to manage data
without writing a single line of SQL
> users.add({ name: 'Rui' }).add({ name: 'Johannes' })
Query OK, 2 items affected (0.0373 sec)
> users.find()
[
{
"_id": "00005b50ced40000000000000001",
"name": "Rui"
},
{
"_id": "00005b50ced40000000000000002",
"name": "Johannes"
}
]
2 documents in set (0.0009 sec)
> users.modify('true').set('team', 'nodejs')
Query OK, 2 items affected (0.0751 sec)
> users.find('name = "Rui"')
[
{
"_id": "00005b50ced40000000000000001",
"name": "Rui",
"team": "nodejs"
}
]
1 document in set (0.0026 sec)
27

Available Methods
• Operational methods return proper database object or operation instances
• CRUD operations: CollectionAdd, CollectionFind, CollectionModify,
CollectionRemove
• Operation boundaries - filtering criteria, computed projections, placeholder
assignment, resultset ordering - established using an expression language
Schema Collection (multi-doc) Collection (single-doc) Session
• createCollection()
• dropCollection()
• add()
• find()
• modify()
• remove()
• createIndex()
• dropIndex()
• getOne()
• addOrReplaceOne()
• replaceOne()
• removeOne()
• startTransaction()
• rollback()
• commit()
• setSavepoint()
• releaseSavepoint()
• rollbackTo()
28

Expression Strings
• Small subset of the SQL language
• Easy to grasp (everyone already knows SQL) but still powerful
• Expressive and human-readable
• Common standard between all the official connector
implementations
// JavaScript
collection
  .find("name = 'foo' AND age > 42")
  .fields("name", "age")
  .groupBy("name", "age")
  .sort("name ASC", "age DESC")
  .limit(4)
  .offset(2)
  .execute()
// Java
collection
  .find("name = 'foo' AND age > 42")
  .fields("name", "age")
  .groupBy("name", "age")
  .sort("name ASC", "age DESC")
  .limit(4)
  .offset(2)
  .execute()
29

Placeholder assignment
• Placeholders are supported by the expression grammar
• Variable values are native to the protocol itself
• Additional server-side validation and escaping
• Mitigates SQL injection risks
30
collection.find('name = :name')
.bind('name', 'foo')
.execute()
 
collection.find('name = :name')
.bind({ name: 'foo' })
.execute()
Mysqlx.Crud.Find {
criteria {
operator {
param {
identifier { document_path: "name" }
}
param { position: 0 }
}
}
args { v_string: "foo" }
}

Raw SQL
• For ETL, structured analytics and/or reporting, SQL is always an
option
• Allows to tap into features not yet part of the X DevAPI, such as:
– DDL operations for relational tables
– Managing key constraints
– Using JOINs
31
// create a table
session.sql('CREATE TABLE foo (bar VARCHAR(3))').execute()
// add a unique constraint
session.sql('ALTER TABLE foo ADD COLUMN bar VARCHAR(3) GENERATED ALWAYS AS doc->>"$.bar" VIRTUAL UNIQUE KEY NOT NULL').execute()
// execute a JOIN query
session.sql('SELECT DISTINCT t1.bar FROM foo t1 JOIN baz t2 ON t1.bar = t2.qux WHERE t1.qux = t2.quux').execute()

Transactions and Savepoints
• Session-level atomic operations
• Create, commit or rollback transactions in the scope of a session
• Create, release or rollback to intermediate savepoints in those
transactions
32
try {
  session.startTransaction()
  // run some operations (1)
session.createSavepoint('foo')
  // run more operations (2)
  session.releaseSavepoint('foo')
  session.commit()
} catch (err) {
  try {
    session.rollbackTo('foo') // go to (2)
  } catch (err) {
   session.rollback() // revert the entire thing
  }
}

Row Locking
• Reads determine isolation level in the presence of concurrent
transactions
• Exclusive and Shared locks with different operation modes:
– DEFAULT (regular SELECT FOR ... UPDATE or SELECT FOR ... SHARE behavior)
– NOWAIT (query executes immediately and fails if row is locked)
– SKIP_LOCKED (query executes immediately and ignores row)
33
// works for table.select() as well
collection.find('name = "foo"').lockExclusive() // mysqlx.LockContention.DEFAULT
collection.find('name = "foo"').lockExclusive(mysqlx.LockContention.NOWAIT)
collection.find('name = "foo"').lockExclusive(mysqlx.LockContention.SKIP_LOCKED)
 
// works for table.select() as well
collection.find('name = "bar"').lockShared() // mysqlx.LockContention.DEFAULT
collection.find('name = "bar"').lockShared(mysqlx.LockContention.NOWAIT)
collection.find('name = "bar"').lockShared(mysqlx.LockContention.SKIP_LOCKED)

MySQL Connector/Node.js
34

Overview
• Only Node.js driver for MySQL with out-of-the-box support for
MySQL 8.0 series and, particularly, the Document Store
• Open source, available on GitHub and on npm
• Modern Node.js asynchronous API using Promises
• Greenfield X Protocol implementation leveraging the X DevAPI spec
• Document Store CRUD API
• Secure connections and multiple authentication mechanisms
• Connection pooling and failover
• Support for raw SQL and relational query builder as well
35

Asynchronous API
• Operations are sent to the server when the execute() method is
called
– receives an optional query operation callback (push-based cursor) which runs
for each element in the result set
– returns a Promise which resolves to a Result instance containing details and
metadata about the operation, or fails with an Error (can be used with
async/await) collection
  .remove()
  .execute(/* no cursor will be needed */)
  .then(result => {
    // result contains details about the operation
  })
  .catch(err => {
    // handle any errors
  })
collection
  .find()
  .execute(doc => {
    // do something with the document
  })
  .then(result => {
    // result contains details about the operation
  })
  .catch(err => {
    // handle any errors
  })
36

Up and running
$ npm install --save --save-exact @mysql/xdevapi
const session = await mysqlx.getSession({ user: 'root' });
const schema = session.getSchema('codeone2018');
 
if (!(await schema.existsInDatabase())) {
await session.createSchema('codeone2018');
}
 
await schema.createCollection('sessions', { ReuseExistingObject: true });
 
const collections = await schema.getCollections();
console.log(collections); // [{ schema: 'codeone2018', collection: 'sessions' }]
const mysqlx = require('@mysql/xdevapi');
37

Adding new documents
// const collection = schema.getCollection('sessions');
await collection.add({ _id: 'DEV5986', title: 'This talk!', speakers: ['Rui Quelhas'] }).execute();
 
const batchA = collection
.add({ _id: 'DEV5985', title: 'MySQL Connector/Node.js and the X DevAPI', speakers: ['Rui Quelhas'] })
.add({ _id: 'DEV6233', title: 'Connector/J Beyond JDBC...', speakers: ['Filipe Silva'] });
 
const batchB = collection
.add([{
_id: 'DEV5959',
title: 'Python and the MySQL Document Store',
interactive: false,
speakers: ['Jesper Krogh'],
topics: ['python']
}, {
_id: 'HOL1706',
title: 'Developing Modern Applications with the MySQL Document Store and Node.js',
interactive: true,
speakers: ['Jesper Krogh', 'Lig Isler-turmelle'],
topics: ['node.js']
}]);
 
await Promise.all([batchA.execute(), batchB.execute()]);
38

Retrieving documents
// const collection = schema.getCollection('sessions'); 
// Retrieving documents with a given criteria
await collection.find('interactive = :interactive OR title LIKE :pattern')
  .bind({ interactive: true, pattern: '%Document Store%' })
  .fields('_id', 'speakers')
  .sort('_id ASC')
  .limit(2)
  .offset(1)
  .execute(console.log); // { _id: 'HOL1706', speakers: ['Jesper Krogh', 'Lig Isler-turmelle'] }
 
// Retrieving a specific document
const doc = await collection.getOne('DEV5985');
console.log(doc); // { _id: 'DEV5985', title: 'MySQL Connector/Node.js and the X DevAPI', speakers: ['Rui Quelhas'] }
 
39

Modifying documents
await collection.modify('title LIKE :pattern AND "Rui Quelhas" IN speakers')
.bind('pattern', '%talk%')
.set('title', 'Node.js and the MySQL Document Store')
.execute();
 
await collection.modify('_id = :id')
.bind('id', 'DEV5986')
.patch({ ongoing: true, topics: ['doc-store'] })
.execute();
 
await collection.modify('ongoing = :ongoing')
.bind('ongoing', true)
.arrayAppend('topics', 'node.js')
.execute();
 
await collection.find('"doc-store" IN topics')
.fields(['ongoing', 'title', 'topics'])
.execute(console.log); // { ongoing: true, title: 'Node.js and the...' topics: ['doc-store', 'node.js'] }
40

Replacing and “Upserting”
 
const existing = 'DEV5959';
 
await collection.replaceOne(existing, { title: 'Python stuff', topics: ['doc-store'] });
 
let talk = await collection.getOne(existing);
console.log(talk); // { _id: 'DEV5959', title: 'Python stuff', topics: ['doc-store'] }
 
await collection.addOrReplaceOne('DEV5959', { interactive: true });
 
const talks = [];
await collection.find('interactive = true')
.fields('_id')
.execute(talk => talks.push(talk));
 
console.log(talks); // [{ _id: 'DEV5959'}, { _id: 'HOL1706' }]
 
41

Removing documents
 
// Remove documents based in a given criteria
await collection.remove('_id LIKE :pattern')
.bind({ pattern: 'DEV%' })
.execute();
 
await collection.find()
.fields(['speakers', 'topics'])
.execute(console.log); // { speakers: ['Jesper Krogh', 'Lig Isler-turmelle'], topics: ['node.js', 'doc-store'] }
 
// Remove a specific document
await collection.removeOne('HOL1706');
 
const doc = await collection.getOne('HOL1706');
console.log(doc); // null
42

ACID
try {
const talks = [];
 
await session.startTransaction();
 
await schema.getCollection('sessions').find('"node.js" IN topics')
.fields('_id')
.lockExclusive(mysqlx.LockContention.DEFAULT) // default value as well
.execute(talk => { talks.push(talk._id); });
 
await schema.createCollection('products', { ReuseExistingObject: true });
 
await schema.getCollection('products').add({ name: 'c-nodejs' }).execute(); 
await schema.getCollection('products').modify('true')
.set('talks', talks)
.execute();
 
await session.commit();
} catch (err) {
await session.rollback();
}
43

Creating secondary indexes
const collection = schema.getCollection('products'); // to comment
await collection.createIndex('name', {
fields: [{
field: '$.name', // column path
required: true,
type: 'TEXT(50)'
}]
});
 
let indexes = [];
 
await session
.sql(`SELECT INDEX_NAME FROM information_schema.STATISTICS WHERE TABLE_NAME = '${collection.getName()}'`)
.execute(index => { indexes = indexes.concat(index); });
 
console.log(indexes); // ['PRIMARY', 'name']
44

Cleaning up
// Delete existing collections
await schema.dropCollection('products');
await schema.dropCollection('sessions');
 
// Delete existing schemas
await session.dropSchema('codeone2018');
 
// Close the server session.
await session.close();
 
45

Some Final Thoughts and Looking Ahead
46

Some takeaways
• Simplify and secure your database installation
• Reap the benefits of both SQL and NoSQL
• High-availability and scalability via InnoDB Cluster
• No secrets or black magic, it’s just the regular JSON datatype
• Fast prototyping but always with maintainability in mind
• Stick to a modern simple Node.js workflow (async, npm)
• Deploy everywhere where Node.js excells (containers,
microservices, serverless and FaaS, desktop, IoT, etc.)
• It’s all open source
47

Looking Ahead
• Protocol level compression
• Prepared statements
• Document linking
• Session connect attributes (app tracing using perf schema)
• Improved relational queries
• And more!
Some areas where we will deliver exciting updates
48

Links and Resources
• https://github.com/mysql/mysql-connector-nodejs
• https://www.npmjs.com/package/@mysql/xdevapi
• https://dev.mysql.com/doc/dev/connector-nodejs/8.0/
• https://ruiquelhas.github.io/percona-live-europe-2017/
• https://www.mysql.com/news-and-events/web-seminars/mysql-
document-store-and-node-js/
• https://www.mysql.com/news-and-events/web-seminars/mysql-
document-store-and-node-js/
• https://insidemysql.com/mysql-document-store-crud-quick-start/
49

Thank you!
50

MySQL Document Store and Node.JS

More Related Content

What's hot (20)

Similar to MySQL Document Store and Node.JS (20)

Recently uploaded (20)

MySQL Document Store and Node.JS